Exploring and exploiting intrinsic synergies of...
Transcript of Exploring and exploiting intrinsic synergies of...
![Page 1: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/1.jpg)
Exploringandexploitingintrinsicsynergiesofmultivariatesystems
FernandoE.RosasResearchFellow
CentreofComplexityScience
DepartmentofMathematics
DepartmentofElectricalandElectronicEngineering
ImperialCollegeLondon
!!!!!Imperial)College)London)))Centre)for)Complexity)Science)
!
!
Complexity
Science
![Page 2: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/2.jpg)
Acknowledges:
Thisworkistheresultofthefortunatecollaborationwith:
!2
PedroA.M.MedianoDepartmentofComputingImperialCollegeLondon HenrikJ.Jensen
DepartmentofMathematicsImperialCollegeLondon
MartinUgarteDepartmentofComputerScienceUniversitéLibreBruxelles
MichaelGastparSchoolofComputerandCommunicationSciencesEPFL
BorzooRassouliSchoolofComputerScienceandElectronicEngineeringUniversityofEssex
DenizGündüzDepartmentofE&EEngineeringImperialCollegeLondon
![Page 3: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/3.jpg)
Acknowledges:MarieSlodowska-CurieActions
!3
Objectives:
(a) tounderstandandexploitintrinsicstatisticalsynergiesofmultivariatesystems
(b) applythistodataprivacyandneuraldataanalysis
![Page 4: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/4.jpg)
!4
Today’smenu
1.Self-organisation
2.O-information
3.Dataprivacy
4.Summaryandcurrentwork
![Page 5: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/5.jpg)
!5
1.Self-organisation:whatisapattern
Whatisapattern/structure?
Shannon—>compressibilityofanstatisticalsourceRegularities/interdepedencies(i.e.deviationsfromstatisticalindependence)
Organisationasstatisticalinterdependency
![Page 6: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/6.jpg)
!6
1.Whatisself-organization?
Attractor:destination/resultoftheevolutionLinearrecurrence/evolutionissimplesimple/uninterestingattractorsNon-linearevolutioninteresting/strangeattractors
Challenge:Isitpossibletorelatetheattractorspropertiestoorganizationproperties?
![Page 7: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/7.jpg)
!7
1.Whatisself-organization?
Attractor:destination/resultoftheevolutionLinearrecurrence/evolutionissimplesimple/uninterestingattractorsNon-linearevolutioninteresting/strangeattractors
Idea:high-ordercorrelationsallowtodistinguishqualitativelydifferentarrangements…
redundancy! synergy!
![Page 8: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/8.jpg)
1.Self-organisation:ourapproach
Approachtostudyspontaneouscreationofcorrelations:
—Consideranflatinitialdistribution(independence)
—Computeit’stimeevolution()usingthecorresponding“masterequation”
—Iterateformanysteps,oruntilitreachastationarystate.
—Analysethe“structureofthecorrelations”of.
Whystartfromanuniformdistribution?Becauseanycorrelationfoundisduetotheevolutionandnottheinitialcondition.
!8
µ0
µt
µtst+1 = µtst
(X1tst , . . . , X
Ntst)
![Page 9: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/9.jpg)
1.Self-organisation:dynamicsassculpture…
!9
“TheAtlas”MichelangeloCirca1530-1534
“TheAwakeningSlave”MichelangeloCirca1520-1523
�t
�0t
![Page 10: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/10.jpg)
1.Self-organisation:entropyisnotenough
Iseasytoprovethat,becauseofdeterminism,thejointShannonentropyofthesystemisnon-increasing.
Thisnotsufficienttoguaranteeself-organization!!(example:fixpointattractors)
!10
H(X1t+1, . . . , X
Nt+1) H(X1
t , . . . , XNt )
InformationisbeingdissipatedIttakesmoreinformationtospecifyarandomstate
thanapointinanattractor
![Page 11: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/11.jpg)
1.Self-organisation:entropyisnotenough
Iseasytoprovethat,becauseofdeterminism,thejointShannonentropyofthesystemisnon-increasing.
Thisnotsufficienttoguaranteeself-organization!!(example:fixpointattractors)
!11
H(X1t+1, . . . , X
Nt+1) H(X1
t , . . . , XNt )
InformationisbeingdissipatedIttakesmoreinformationtospecifyarandomstate
thanapointinanattractor
Keyidea:Thedestructionofinformation
cancreatecorrelations!!
H(X) H(Y )
H(X) H(Y )
I(X;Y )
![Page 12: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/12.jpg)
1.Self-organisation:proposedframework
!12
Ourproposedtotalentropydecomposition:
F.Rosas,P.A.M.Mediano,M.Ugarte,H.J.Jensen,“Aninformation-theoreticapproachtoself-organization:emergenceofcomplexinterdependenciesincoupleddynamicalsystems”,Entropy20,no.10(2018):793.
"Residualentropy”
“Bindingentropy”
![Page 13: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/13.jpg)
1.Self-organisation:proposedframework
Decompositionbysharingmodes:
!13
bymeasuringiagentscanguessthestateofn…
![Page 14: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/14.jpg)
1.Self-organisation:proposedframework
Decompositionbysharingmodes:
where
!14
![Page 15: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/15.jpg)
1.Self-organisation:proofofconcept
Rule232
!15
![Page 16: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/16.jpg)
1.Self-organisation:proofofconcept
Rule232
!16
![Page 17: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/17.jpg)
1.Self-organisation:proofofconcept
Rule30
!17
![Page 18: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/18.jpg)
1.Self-organisation:proofofconcept
Rule30
!18
![Page 19: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/19.jpg)
1.Self-organisation:proofofconcept
!19
![Page 20: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/20.jpg)
1.Self-organisation:proofofconcept
!20
0 5 10 15
0.0
0.5
1.0
yL
Rule 232: Redundancy
0 5 10 15
0.0
0.5
1.0
Rule 30: Moderate synergy
0 5 10 15
0.0
0.5
1.0
Number of cells (L)
yL
Rule 90: Strong synergy
0 5 10 15
0.0
0.5
1.0
Number of cells (L)
Rule 106: Mixed profile
Rule106
![Page 21: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/21.jpg)
!21
Today’smenu
1.Self-organisation
2.O-information
3.Dataprivacy
4.Summaryandcurrentwork
![Page 22: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/22.jpg)
2.O-information:fundamentals
1.Interaction-information:
2.Totalcorrelation:
3.Dualtotalcorrelation:
Stateofaffairs:
(+)TCandDTCaremetricsofglobalcorrelationstrength.
(-)TC=0ifandonlyifDTC=0.Besidesthat,theirrelationshipisunclear.
(+)
(+)=TC-DTC.
(-)Themeaningoftheinteraction-informationforn>3isunclear.
!22
ispositiveforredundantsystems,andnegativeforsynergisticones.I(X1;X2;X3) = I(X1;X3) + I(X2;X3)� I(X1X2;X3)
I(X1;X2;X3)
TC(Xn) =nX
j=1
H(Xj)�H(Xn)
DTC(Xn) = H(Xn)�nX
j=1
H(Xj |Xn�j)
I(X1;X2; . . . ;Xn) = �X
�✓{1,...,n}
(�1)|�|H(X�)
![Page 23: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/23.jpg)
2.O-information:fundamentals
1.Interaction-information:
2.Totalcorrelation:
3.Dualtotalcorrelation:
!23
HowcouldI“fix”theinteraction-infofor
anyn??
TCorDTC?thatisthequestion…
Theinformation-theoreticHamlet
TC(Xn) =nX
j=1
H(Xj)�H(Xn)
DTC(Xn) = H(Xn)�nX
j=1
H(Xj |Xn�j)
I(X1;X2; . . . ;Xn) = �X
�✓{1,...,n}
(�1)|�|H(X�)
![Page 24: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/24.jpg)
2.O-information:thetwofacesofinterdependency
Considerasystemwhere,describedbyap.d.f..
!24
nX
j=1
log |Xj |
N (Xn) =X
j
log |Xj |�H(Xn)
H(Xn) = �X
xn
pxn log pxn
Negentropy
Entropy
Xn = (X1, . . . , Xn) Xj 2 Xj pXn(xn)
H(Xn)
N (Xn)
![Page 25: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/25.jpg)
2.O-information:thetwofacesofinterdependency
Considerasystemwhere,describedbyap.d.f..
!25
nX
j=1
log |Xj |
Negentropy
Entropy
X1, X2 2 {0, 1}
pX1X2(0, 0) = pX1X2(1, 1) = 0
pX1X2(0, 1) = pX1X2(1, 0) = 1/2
H(X1X2) = N (X1X2) = 1
(0, 0) (0, 1)
(1, 0) (1, 1)+
+
Example:
H(Xn) = �X
xn
pxn log pxn
N (Xn) =X
j
log |Xj |�H(Xn)
Xn = (X1, . . . , Xn) Xj 2 Xj pXn(xn)
H(Xn)
N (Xn)
![Page 26: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/26.jpg)
2.O-information:thetwofacesofinterdependency
!26
N (X5)N (X4)N (X3)N (X2)N (X1)
TC
N (X6)
nX
j=1
log |Xj |
H(Xn) = �X
xn
pxn log pxn
N (Xn) =nX
j=1
N (Xj) + TC
IndividualconstraintsCollectiveconstraints
Marginalnegentropy: N (Xj) = log |Xj |�H(Xj)
Collectiveconstraints: TC(Xn) = N (Xn)�nX
j=1
N (Xj) =nX
j=1
H(Xj)�H(Xn)
![Page 27: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/27.jpg)
2.O-information:thetwofacesofinterdependency
!27
N (X5)N (X4)N (X3)N (X2)N (X1)
TC
N (X6)
nX
j=1
log |Xj |
H(Xn) = �X
xn
pxn log pxn
N (Xn) =nX
j=1
N (Xj) + TC
(0, 0) (0, 1)
(1, 0) (1, 1)+
+Example: (0, 0) (0, 1)
(1, 0) (1, 1)+ +
N (X1X2) = 1
TC(X1X2) = 1
N (X1X2) = 1
TC(X1X2) = 0
IndividualconstraintsCollectiveconstraints
![Page 28: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/28.jpg)
2.O-information:thetwofacesofinterdependency
!28
R1R2 R3 R4 R5
N (X5)N (X4)N (X3)N (X2)N (X1)
TC
DTC
N (X6)
R6
N (Xn) =nX
j=1
N (Xj) + TC
H(Xn) =nX
j=1
Rj +DTC
nX
j=1
log |Xj |
Privaterandomness
Collectiveconstraints
Sharedrandomness
Individualconstraints
Residualentropies:
Bindingentropy: DTC(Xn) = H(Xn)�nX
j=1
H(Xj |Xn�j)
Rj = H(Xj |Xn�j)
![Page 29: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/29.jpg)
2.O-information:thetwofacesofinterdependency
!29
R1R2 R3 R4 R5
N (X5)N (X4)N (X3)N (X2)N (X1)
TC
DTC
N (X6)
R6
N (Xn) =nX
j=1
N (Xj) + TC
H(Xn) =nX
j=1
Rj +DTC
nX
j=1
log |Xj |
Privaterandomness
Collectiveconstraints
Sharedrandomness
Example: X1 = X2 = X3
TC = 2 > DTC = 1
X3 = X1 �X2
TC = 1 < DTC = 2
redundancy! synergy!
Individualconstraints
![Page 30: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/30.jpg)
2.O-information:thetwofacesofinterdependency
!30
R1R2 R3 R4 R5
N (X5)N (X4)N (X3)N (X2)N (X1)
TC
DTC
N (X6)
R6
N (Xn) =nX
j=1
N (Xj) + TC
H(Xn) =nX
j=1
Rj +DTC
nX
j=1
log |Xj |
Privaterandomness
Collectiveconstraints
Sharedrandomness
Example: X1 = X2 = X3
TC = 2 > DTC = 1
X3 = X1 �X2
TC = 1 < DTC = 2
redundancy! synergy!
Individualconstraints
![Page 31: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/31.jpg)
2.O-information:definition
Occam’sraizor(lexparsimoniae):givepreferencetothesimplestdescription
• Ifitisshortertodescribetheallowedstates.
• Ifitisshortershortertodescribetheconstraints.
!31
Definition
⌦(Xn) = TC(Xn)�DTC(Xn)O-information:
⌦(Xn) > 0
⌦(Xn) < 0
F.Rosas,P.A.M.Mediano,M.Gastpar,andH.J.Jensen."QuantifyingHigh-orderInterdependenciesviaMultivariateExtensionsoftheMutualInformation,”accepted,tobepublishedinPRE,2019
![Page 32: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/32.jpg)
2.O-information:definition
Occam’sraizor(lexparsimoniae):givepreferencetothesimplestdescription
• Ifitisshortertodescribetheallowedstates.
• Ifitisshortershortertodescribetheconstraints.
!32
Definition
⌦(Xn) = TC(Xn)�DTC(Xn)O-information:
⌦(Xn) > 0
⌦(Xn) < 0
Example:
redundancy! synergy!
X1 = X2 = · · · = Xn
TC = n� 1 > DTC = 1 TC = 1 < DTC = n� 1
Xn = X1 �X2 � · · ·�Xn�1
F.Rosas,P.A.M.Mediano,M.Gastpar,andH.J.Jensen."QuantifyingHigh-orderInterdependenciesviaMultivariateExtensionsoftheMutualInformation,”accepted,tobepublishedinPRE,2019
![Page 33: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/33.jpg)
2.O-information:definition
Occam’sraizor(lexparsimoniae):givepreferencetothesimplestdescription
• Ifitisshortertodescribetheallowedstates.
• Ifitisshortershortertodescribetheconstraints.
!33
Definition
⌦(Xn) = TC(Xn)�DTC(Xn)O-information:
⌦(Xn) > 0
⌦(Xn) < 0
Definition
1. Asystemisredundancy-dominatedif
2. Asystemissynergy-dominatedif
⌦(Xn) � 0
⌦(Xn) 0
![Page 34: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/34.jpg)
2.O-information:definition
Occam’sraizor(lexparsimoniae):givepreferencetothesimplestdescription!
• Ifitisshortertodescribetheallowedstates.
• Ifitisshortershortertodescribetheconstraints.
Note:
1.n=2:,becausei.e.sharedrandomnessisequaltopredictability
!34
Definition
⌦(Xn) = TC(Xn)�DTC(Xn)O-information:
⌦(Xn) > 0
⌦(Xn) < 0
⌦(X1X2) = 0 TC(X1X2) = DTC(X1X2) = I(X1;X2)
![Page 35: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/35.jpg)
2.O-information:definition
Occam’sraizor(lexparsimoniae):givepreferencetothesimplestdescription!
• Ifitisshortertodescribetheallowedstates.
• Ifitisshortershortertodescribetheconstraints.
Note:
1.n=2:,becausei.e.sharedrandomnessisequaltopredictability
2.n=3:—>interaction-information!
!35
Definition
⌦(Xn) = TC(Xn)�DTC(Xn)O-information:
⌦(Xn) > 0
⌦(Xn) < 0
⌦(X1X2) = 0 TC(X1X2) = DTC(X1X2) = I(X1;X2)
⌦(X1X2X3) = I(X1;X2)� I(X1;X2|X3) =: I(X1;X2;X3)
![Page 36: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/36.jpg)
2.O-information:(1)itissumoftripleinteraction-informations
• DecompositionsfortheTC,DTCando-information:
!36
O-informationisanaggregationoftriplemulti-informations!!
![Page 37: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/37.jpg)
2.O-information:(1)itissumoftripleinteraction-informations
• DecompositionsfortheTC,DTCando-information:
!37
![Page 38: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/38.jpg)
2.O-information:(1)itissumoftripleinteraction-informations
• DecompositionsfortheTC,DTCando-information:
!38
HXY ZW
HXY +HZW
HXZ +HYW
HXW +HY Z
HY +HZWXHX +HY ZW HZ +HWXY HW +HXY Z
HY +HZ +HXW
HX +HW +HY ZHX +HZ +HYWHX +HY +HZW HY +HW +HXZ HZ +HW +HXY
HX +HY +HZ +HW
HXY |ZW +HZW |XY
HXZ|YW +HYW |XZ
HXW |Y Z +HY Z|XW
HY |ZWX +HZWX|YHX|Y ZW +HY ZW |X HZ|WXY +HWXY |Z HW |XY Z +HXY Z|W
HY |ZWX +HZ|WXY +HXW |Y Z
HX|Y ZW +HW |XY Z +HY Z|XW
HX|Y ZW +HZ|WXY +HYW |XZHX|Y ZW +HY |ZWX +HZW |XY HY |ZWX ++HW |XY Z +HXZ|YW HZ|WXY +HW |XY Z +HXY |ZW
HX|Y ZW +HY |ZWX +HZ|WXY +HW |XY Z
![Page 39: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/39.jpg)
2.O-information:(2)characterizationofextremevalues
• DecompositionsfortheTC,DTCando-information:
• Upperandlowerbounds:
• Characterizationofuniqueextremesoftheo-information:
!39
Maximum: X1 = X2 = · · · = Xn Minimum: Xn =n�1X
j=1
Xj (mod k)
![Page 40: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/40.jpg)
2.O-information:(3)nullvalueofnon-overlappingpairwiseinteractions
• O-informationofindependentsubgroupsisadditive:
• Ifasystem’sisdecomposableindisjointpairwiseinteractionsthen
• Theconverseisnottrue!!(synergiesandredundanciescanceleachother)
• Localo-informationcangiveafine-graineddescriptionofthesystem…
!40
⌦(Xn,Y m) = ⌦(Xn) + ⌦(Y m)pXn,Y m(xn,ym) = pXn(xn)pY m(ym)
⌦(Xn) = 0
!i,j = I(Xi;Xj ;Xn�i�j)
![Page 41: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/41.jpg)
2.O-information:(4)valueimpliesboundsoverdifferentscales
• ThevalueoftheO-informationprovideconstraintsovertheinterdependenciesofsubgroups!
•
!41
![Page 42: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/42.jpg)
2.O-information:(5)redundancyoftrees
• O-informationisnon-negativeingraphicalmodelswiththreestructure!
• ForNaiveBayes,theo-informationisgivenby
• ForMarkovchains,theo-informationis
!42
⌦(Xn) = C(Xn2 ) � 0
1 2 n…3
![Page 43: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/43.jpg)
2.O-information:(6)synergyispervasiveinlargesystems
• Themeanvalueoftheo-informationoverrandomdistributionsgrowsnegative!
!43
2 4 6 8
�4
�2
0
System size n
Gaussian systems
2 4 6 8
�0.8
�0.6
�0.4
�0.2
0
System size n
⌦
Binary systems
AnalyticalMonte CarloApproximation
(Williams & Beer, 2010)(Williams & Beer, 2010)(Williams & Beer, 2010)(Williams & Beer, 2010)
![Page 44: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/44.jpg)
2.O-information:relationshipwithstatisticalmechanics
1.Connectedinformation(Schneidmanetal.2003):
Stateofaffairs:
(+)Interestingconnectionwithstatisticalmechanics.
(+)Intuitiveinterpretation.
(-)Veryhardtocompute…
!44
Con(Xn) =
Pnk=1 H(Xj)�H(Xn
maxent)
TC(Xn)
![Page 45: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/45.jpg)
2.O-information:relationshipwithstatisticalmechanics
1.Connectedinformation(Schneidmanetal.2003):
• Hamiltonianwithhigh-ordertermstendtohavenegativeO-information
!45
Con(Xn) =
Pnk=1 H(Xj)�H(Xn
maxent)
TC(Xn)
![Page 46: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/46.jpg)
2.O-information:analysispipeline
!46
Procedure
1.Computeandasmetricsofglobalcorrelationstrength.
2.Computetofinddominantglobalbehaviour(redundancyorsynergy).
3.Studythelocalo-informationterms,i.e.foralliandj,asameasureoflocalisedbehaviour.
⌦(Xn)
I(Xi;Xj ;Xn�i,�j)
B(Xn) C(Xn)
![Page 47: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/47.jpg)
2.O-information:Casestudyonbaroquemusicscores
DataanalysisovermusicscoresfromtheBaroqueperiod(Python,Music21package)
i)choralesforfourvoicesbyJ.S.Bach(1685–1750)
!47
![Page 48: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/48.jpg)
2.O-information:Casestudyonbaroquemusicscores
DataanalysisovermusicscoresfromtheBaroqueperiod(Python,Music21package)
i)choralesforfourvoicesbyJ.S.Bach(1685–1750)
Dataformat:foursynchronoustimeseries,alphabetof13values(12tones+silence)
Database:300~chorales,43kfour-notechords
!48
![Page 49: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/49.jpg)
2.O-information:Casestudyonbaroquemusicscores
DataanalysisovermusicscoresfromtheBaroqueperiod(Python,Music21package)
i)choralesforfourvoicesbyJ.S.Bach(1685–1750)
(43kfour-notechords)
ii)Op.1,3,4,5and6ofA.Corelli(1653–1713)(80kfour-notechords)
!49
![Page 50: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/50.jpg)
2.O-information:Casestudyonbaroquemusicscores
DataanalysisovermusicscoresfromtheBaroqueperiod(Python,Music21package)
!50
![Page 51: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/51.jpg)
2.O-information:Casestudyonbaroquemusicscores
DataanalysisovermusicscoresfromtheBaroqueperiod(Python,Music21package)
!51
Synergyallowsthecoexistenceoflocalindependencyandglobalcoordination
![Page 52: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/52.jpg)
!52
Today’smenu
1.Self-organisation
2.O-information
3.Dataprivacy
4.Summaryandcurrentwork
![Page 53: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/53.jpg)
3.Synergyanddataprivacy:keyideas
Privacyfunnel:maximizecorrelationwithfeatureofinterest,whilekeepingprivatestuffsecure
!53
![Page 54: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/54.jpg)
3.Synergyanddataprivacy:keyideas
Privacyfunnel:maximizecorrelationwithfeatureofinterest,whilekeepingprivatestuffsecure
However,itisoftenthecasewherethefeatureofinterestisunknown
ButtheDPinequalitymakeitseemunfeasible…
!54
![Page 55: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/55.jpg)
3.Synergyanddataprivacy:keyideas
Idea:makeYindependentofeachcoordinateofX,butcorrelatedwiththewhole!!
!55
![Page 56: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/56.jpg)
3.Synergyanddataprivacy:keyideas
Idea:makeYindependentofeachcoordinateofX,butcorrelatedwiththewhole!!
!56
![Page 57: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/57.jpg)
3.Synergyanddataprivacy:mainresults
!57
![Page 58: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/58.jpg)
3.Synergyanddataprivacy:mainresults
!58
Tolearnmoreaboutthis:1. B.Rassouli,F.E.Rosas,andD.Gunduz."DataDisclosureunderPerfectSamplePrivacy."Submitted
toIEEETransactionsinInformationForensicsandSecurity(TIFS),underreview.arXivpreprintarXiv:1904.01711(2019).
2. B.Rassouli*,F.E.Rosas*,andD.Gunduz,"LatentFeatureDisclosureunderPerfectSamplePrivacy."In2018IEEEInternationalWorkshoponInformationForensicsandSecurity(WIFS),pp.1-7.IEEE,2018.
![Page 59: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/59.jpg)
!59
Today’smenu
1.Self-organisation
2.O-information
3.Dataprivacy
4.Summaryandcurrentwork
![Page 60: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/60.jpg)
Furtherreading:
!60
1. B.Rassouli*,F.E.Rosas*,andD.Gunduz,"LatentFeatureDisclosureunderPerfectSamplePrivacy."In2018IEEEInternationalWorkshoponInformationForensicsandSecurity(WIFS),pp.1-7.IEEE,2018.
2. B.Rassouli,F.E.Rosas,andD.Gunduz."DataDisclosureunderPerfectSamplePrivacy."SubmittedtoIEEETransactionsinInformationForensicsandSecurity(TIFS),underreview.arXivpreprintarXiv:1904.01711(2019).
3. F.Rosas,P.A.M.Mediano,M.Ugarte,H.J.Jensen,“Aninformation-theoreticapproachtoself-organization:emergenceofcomplexinterdependenciesincoupleddynamicalsystems”,Entropy20,no.10(2018):793.
4. F.Rosas,P.A.M.Mediano,M.Gastpar,andH.J.Jensen."QuantifyingHigh-orderInterdependenciesviaMultivariateExtensionsoftheMutualInformation,”accepted,tobepublishedinPRE,2019
![Page 61: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/61.jpg)
PD:currentwork
Drivingquestions:1.Isthebrainasynergisticmulti-agentsystem?2.Isconsciousnessrelatedwithhigh-orderstatistics?3.CanPIDprincipleshelpustounderstandtheeffectofpsychedelicdrugs?
!61
!!!!!Imperial)College)London)))Centre)for)Complexity)Science)
!
!
Complexity
Science
![Page 62: Exploring and exploiting intrinsic synergies of ...csc.ucdavis.edu/~chaos/share/frosas_talk_beyondshannon.pdfExploring and exploiting intrinsic synergies of multivariate systems Fernando](https://reader034.fdocuments.in/reader034/viewer/2022042401/5f0fee117e708231d4469806/html5/thumbnails/62.jpg)
Acknowledges:
Thisworkistheresultofthefortunatecollaborationwith
- PedroMediano(ImperialCollegeLondon)- HenrikJ.Jensen(ImperialCollegeLondon)- MichaelGastpar(EPFL)- MartinUgarte(UniversitélibredeBruxelles)- BorzooRasouli(UniversityofEssex)- DenizGündüz(ImperialCollegeLondon)
SpecialthanksfortheMarieSlodowska-CurieprogramfromH20202andtheEuropeanCommission.
!62