Statistics IV
July 22, 2020
来嶋 秀治 (Shuji Kijima)
Dept. Informatics,
Graduate School of ISEE
Todays topics
• linear regression (線形回帰)
•単回帰
• 重回帰
•自己回帰
• モデル選択 AIC
確率統計特論 (Probability & Statistics)
Lesson 11
2
Final exam PLAN (期末試験案)
Date/time: August 5 or 12 (8/5 or 12), 13:00-
Place (場所): moodle.
Submit electric file (incl. photo).
電子ファイルを提出 (手書きを写真にとって提出可).
Topics (範囲):
Probability and Statistics.
check the course page (講義ページを参照のこと)
http://tcs.inf.kyushu-u.ac.jp/~kijima/
Books, notes, google, etc. are allowed to use (持ち込み可).
Communication (e-mail, SNS, BBS) is prohibited.
What do you prefer? Please click. 3
1. Final exam on August 5:
期末試験は8/5にしてほしい.
2. Final exam on August 12: Day-off on August 5.
期末試験は8/12にして、8/5は休講にしてほしい.
3. Final exam on August 12: Advanced topic on August 5.
期末試験は8/12にして、8/5は発展的話題を聞きたい.
Statistics Inference (統計的推論)
Estimation (推定)
Statistical test (統計検定)
Regression (回帰)
Correlation (相関)
Time series analysis (時系列解析)
Classification/Clustering (分類)
Applications
Machine learning (機械学習),
Pattern recognition (パターン認識),
Data mining (データマイニング), etc.
Statistics / Data science4
Statistics Inference (統計的推論)
Estimation (推定) ←July 1,8 and next week
Statistical test (統計検定) ←last week
Regression (回帰) ←today
Statistics / Data science5
Linear regression
Ex. Advertisement7
Question
How does 𝑦 increase, as 𝑥 increasing?
year 1 2 3 4 5 6 7 8
𝑥: ad. cost 8 11 13 10 15 19 17 20
𝑦: sale amount 115 124 138 120 151 186 169 193
Ex. Advertisement8
Question
How does 𝑦 increase, as 𝑥 increasing?
year 1 2 3 4 5 6 7 8
𝑥: ad. cost 8 11 13 10 15 19 17 20
𝑦: sale amount 115 124 138 120 151 186 169 193
0
50
100
150
200
250
0 5 10 15 20 25
系列1
Least Square Estimator9
Question
How does 𝑦 increase, as 𝑥 increasing?
Linear regression (線形回帰)
Suppose 𝑦𝑖 = 𝛼 + 𝛽𝑥𝑖 + 𝑒𝑖 where 𝑒𝑖 ∼ N(0, 𝜎2).
Estimate 𝛼 and 𝛽 such that
min
𝑖=1
𝑛
𝑦𝑖 − 𝛼 + 𝛽𝑥𝑖2
year 1 2 3 4 5 6 7 8
𝑥: ad. cost 8 11 13 10 15 19 17 20
𝑦: sale amount 115 124 138 120 151 186 169 193
Least Square Estimator10
Linear regression (線形回帰)
Suppose 𝑦𝑖 = 𝛼 + 𝛽𝑥𝑖 + 𝑒𝑖 where 𝑒𝑖 ∼ N(0, 𝜎2).
Estimate 𝛼 and 𝛽 such that minσ𝑖=1𝑛 𝑦𝑖 − 𝛼 + 𝛽𝑥𝑖
2
𝜕
𝜕𝛼𝑔 𝛼, 𝛽 =
𝑖=1
𝑛
−2(𝑦𝑖 − (𝛼 + 𝛽𝑥𝑖))
𝜕
𝜕𝛽𝑔 𝛼, 𝛽 =
𝑖=1
𝑛
(−2𝑥𝑖)(𝑦𝑖 − (𝛼 + 𝛽𝑥𝑖))
መ𝛽 =ො𝛼 =
Least Square Estimator11
Linear regression (線形回帰)
Suppose 𝑦𝑖 = 𝛼 + 𝛽𝑥𝑖 + 𝑒𝑖 where 𝑒𝑖 ∼ N(0, 𝜎2).
Estimate 𝛼 and 𝛽 such that minσ𝑖=1𝑛 𝑦𝑖 − 𝛼 + 𝛽𝑥𝑖
2
𝜕
𝜕𝛼𝑔 𝛼, 𝛽 =
𝑖=1
𝑛
−2(𝑦𝑖 − (𝛼 + 𝛽𝑥𝑖))
𝜕
𝜕𝛽𝑔 𝛼, 𝛽 =
𝑖=1
𝑛
(−2𝑥𝑖)(𝑦𝑖 − (𝛼 + 𝛽𝑥𝑖))
𝛼 + 𝛽 ҧ𝑥 = ത𝑦
𝛼 ҧ𝑥 + 𝛽𝑥2 = 𝑥𝑦𝜕
𝜕𝛽𝑔 𝛼, 𝛽 = 0
𝜕
𝜕𝛼𝑔 𝛼, 𝛽 = 0
መ𝛽 =𝑥𝑦 − ҧ𝑥 ⋅ ത𝑦
𝑥2 − ҧ𝑥2
ො𝛼 = ത𝑦 − መ𝛽 ҧ𝑥
Ex. Advertisement12
year 1 2 3 4 5 6 7 8
𝑥: ad. cost 8 11 13 10 15 19 17 20
𝑦: sale amount 115 124 138 120 151 186 169 193
𝑥 ≔1
𝑛
𝑖=1
𝑛
𝑥𝑖 =113
8= 14.125
𝑦 ≔1
𝑛
𝑖=1
𝑛
𝑦𝑖 =1196
8= 149.5
𝑥2 ≔1
𝑛
𝑖=1
𝑛
𝑥𝑖2 =
1729
8= 216.25
𝑥𝑦 ≔1
𝑛
𝑖=1
𝑛
𝑥𝑖𝑦𝑖 =17810
8= 2226.25
መ𝛽 =𝑥𝑦 − 𝑥 ⋅ 𝑦
𝑥2 − 𝑥2
=2226.25 − 14.125 × 149.5
216.125 − 14.1252= 6.9
ො𝛼 = 𝑦 − መ𝛽𝑥 = 149.5 − 6.9 × 14.125 = 52.1
E መ𝛽 = E𝑥𝑦 − ҧ𝑥 ത𝑦
𝑥2 − ҧ𝑥2=?
E ො𝛼 = E 𝑦 − መ𝛽 ҧ𝑥 =?
Q: Are ො𝛼, መ𝛽 unbiased estimators?13
E መ𝛽 = E𝑥𝑦 − ҧ𝑥 ത𝑦
𝑥2 − ҧ𝑥2= 𝛽
E ො𝛼 = E 𝑦 − መ𝛽 ҧ𝑥 = 𝛼
Q: Are ො𝛼, መ𝛽 unbiased estimators?14
?
?
E መ𝛽 = E𝑥𝑦 − ҧ𝑥 ത𝑦
𝑥2 − ҧ𝑥2=E 𝑥𝑦 − ҧ𝑥E ത𝑦
𝑥2 − ҧ𝑥2= 𝛽
E ො𝛼 = E 𝑦 − መ𝛽 ҧ𝑥 = E ത𝑦 − ҧ𝑥E መ𝛽 = 𝛼
Q: Are ො𝛼, መ𝛽 unbiased estimators?15
?
?
E መ𝛽 = E𝑥𝑦 − ҧ𝑥 ത𝑦
𝑥2 − ҧ𝑥2=E 𝑥𝑦 − ҧ𝑥E ത𝑦
𝑥2 − ҧ𝑥2= 𝛽
E ො𝛼 = E 𝑦 − መ𝛽 ҧ𝑥 = E ത𝑦 − ҧ𝑥E መ𝛽 = 𝛼
Q: Are ො𝛼, መ𝛽 unbiased estimators?16
?
?
E 𝑦 =1
𝑛
𝑖=1
𝑛
E 𝑦𝑖 =1
𝑛
𝑖=1
𝑛
𝛼 + 𝛽𝑥𝑖 = 𝛼 + 𝛽𝑥
E 𝑥𝑦 =1
𝑛
𝑖=1
𝑛
E 𝑥𝑖𝑦𝑖 =1
𝑛
𝑖=1
𝑛
𝑥𝑖 𝛼 + 𝛽𝑥𝑖 = 𝛼𝑥 + 𝛽𝑥2
E መ𝛽 =E 𝑥𝑦 − ҧ𝑥E ത𝑦
𝑥2 − ҧ𝑥2=𝛼𝑥 + 𝛽𝑥2 − ҧ𝑥 𝛼 + 𝛽𝑥
𝑥2 − ҧ𝑥2= 𝛽
E ො𝛼 = E ത𝑦 − ҧ𝑥E መ𝛽 = 𝛼 + 𝛽 ҧ𝑥 − ҧ𝑥𝛽 = 𝛼
Variance (Gauss–Markov theorem)17
Thus
ො𝛼 ∼ N 𝛼, 𝑎 𝑥 𝜎2
መ𝛽 ∼ N(𝛽, 𝑏 𝑥 𝜎2)
Thm.
Let ො𝜎2: =1
𝑛−2σ𝑖=1𝑛 𝑦𝑖 − ො𝛼 + መ𝛽𝑥𝑖
2,
then 𝐸 ො𝜎2 = 𝜎2 and 𝑛−2 ෝ𝜎2
𝜎2∼ 𝜒𝑛−2
2 hold.
omit the proof (not easy)
Remark
Var ෝ𝛼 and Var 𝛽 decrease
as 𝑠𝑥2 =
σ𝑖=1𝑛 (𝑥𝑖
2− 𝑥 2)
𝑛increases.
Observe 𝑥 in a wide range,
then we obtain a good estimator.
Var ො𝛼 ≔ E ො𝛼 − 𝛼 2 =1
𝑛1 +
𝑥2
𝑠𝑥2 𝜎2
Var መ𝛽 ≔ E መ𝛽 − 𝛽2=
1
𝑛⋅𝑠𝑥2 𝜎
2
Cf. 中田,内藤「確率・統計」
Hypothesis testing
Least Square Estimator19
Question
Does the data support the claim 𝛽 = 0?
Linear regression (線形回帰)
Suppose 𝑦𝑖 = 𝛼 + 𝛽𝑥𝑖 + 𝑒𝑖 where 𝑒𝑖 ∼ N(0, 𝜎2).
Estimate 𝛼 and 𝛽 such that
min
𝑖=1
𝑛
𝑦𝑖 − 𝛼 + 𝛽𝑥𝑖2
year 1 2 3 4 5 6 7 8
𝑥: ad. cost 8 11 13 10 15 19 17 20
𝑦: sale amount 115 124 138 120 151 186 169 193
Hypothesis testing for 𝛽20
The central limit theorem suggests that መ𝛽 − 𝛽
Var መ𝛽
∼ N 0,1
Then, its Studentization is
𝑇𝑛 ≔መ𝛽 − 𝛽
𝜎2
𝑛 ⋅ 𝑠𝑥2
≃መ𝛽 − 𝛽
Var መ𝛽
Thm.
𝑇𝑛 ∼ 𝑡𝑛−2
Ex. Advertisement21
𝑡6−2∗ = 2.477 null hypothesis 𝛽 = 0 is rejected.
year 1 2 3 4 5 6 7 8
𝑥: ad. cost 8 11 13 10 15 19 17 20
𝑦: sale amount 115 124 138 120 151 186 169 193
መ𝛽 =𝑥𝑦 − 𝑥 ⋅ 𝑦
𝑥2 − 𝑥2
= 6.9
ො𝛼 = 𝑦 − መ𝛽𝑥 = 52.1
𝜎2 =1
𝑛 − 2
𝑖=1
𝑛
𝑦𝑖 − ො𝛼 + መ𝛽𝑥𝑖2
= 21.418
𝑇𝑛 =መ𝛽 − 𝛽
𝜎2
𝑛 ⋅ 𝑠𝑥2
=6.9 − 0
4.6282
132.875
= 17.19
Ex.
Least Square Estimator23
1 2 3 … … 37 38
x: applied dose
(投与量)
2.32 2.39 2.61 7.78 8.28
y: observed value
(観測数値)
2.88 3.21 3.01 5.88 6.67
Question
How does 𝑦 increase, as 𝑥 increasing?
Linear regression (線形回帰)
Suppose 𝑦𝑖 = 𝛼 + 𝛽𝑥𝑖 + 𝑒𝑖 where 𝑒𝑖 ∼ N(0, 𝜎2).
Estimate 𝛼 and 𝛽 such that
min
𝑖=1
𝑛
𝑦𝑖 − 𝛼 + 𝛽𝑥𝑖2
ex24
ො𝛼 = 2.07 , መ𝛽 = 0.49 , ො𝜎2 = 0.472, 𝑠𝑥2 = ⋯
Q:𝛽 = 0?
administration does not work? (投与効果はない?)
If 𝑇𝑛 > |𝑡36∗ | then
the null hypothesis 𝛽 = 0 is rejected.
𝑇𝑛 ≔መ𝛽 − 𝛽
𝜎2
𝑛 ⋅ 𝑠𝑥2
=0.49 − 0
0.472
38 ×? ? ?
= ⋯
Multiple Linear Regression
Least Square Estimator26
Proposition
The optimum solution is 𝜷 = 𝑋⊤𝑋−1𝑋⊤𝒚.
Furthermore, 𝜷 is an unbiased estimator.
where 𝒚 =𝑦1⋮𝑦𝑛
and 𝑿 =𝒙𝟏⊤
⋮𝒙𝒏⊤
Multiple linear regression (多重線形回帰)
Suppose that 𝑦𝑖 = 𝒙𝒊⊤𝜷 + 𝑒𝑖 and 𝑒𝑖 ∼ N(0, 𝜎2).
Estimate 𝜷 such that
min𝜷
𝑖=1
𝑛
𝑦𝑖 − 𝒙𝒊⊤𝜷
2= min
𝜷𝒚 − 𝑋𝜷 ⊤(𝒚 − 𝑋𝜷)
Multi linear regression27
Proposition
The optimum solution is 𝜷 = 𝑋⊤𝑋−1𝑋⊤𝒚.
𝛻 𝒚 − 𝑋𝜷 ⊤ 𝒚 − 𝑋𝜷 = −2𝑋⊤ 𝒚 − 𝑋𝜷
𝑋⊤𝒚 − 𝑋⊤𝑋𝜷 = 0
𝑋⊤𝑋𝜷 = 𝑋⊤𝒚
Rem 10-1. (see Apex.)
𝛻 𝐴𝒙 ⊤ 𝐴𝒙 = 2𝐴⊤𝐴𝒙
Apex. Proof of Rem. 10-128
Rem 10-1. (see Apex.)
𝛻 𝐴𝒙 ⊤ 𝐴𝒙 = 2𝐴⊤𝐴𝒙
Since
𝜕 𝑎𝑖1𝑥1 +⋯+ 𝑎𝑖𝑑𝑥𝑑2
𝜕𝑥𝑗= 2 𝑎𝑖1𝑥1 +⋯+ 𝑎𝑖𝑑𝑥𝑑 𝑎𝑖𝑗
we have
𝛻 𝑎𝑖1𝑥1 +⋯+ 𝑎𝑖𝑑𝑥𝑑2 = 2(𝑎𝑖1𝑥1, … , 𝑎𝑖𝑑𝑥𝑑)
𝑎𝑖1⋮𝑎𝑖𝑑
= 2 𝒂𝒊⊤𝒙 𝒂𝒊
Then,
𝑖=1
𝑑
𝛻 𝑎𝑖1, … , 𝑎𝑖𝑑2 = 2
𝑖=1
𝑑
𝒂𝒊⊤𝒙 𝒂𝒊 = 2 𝒂𝟏… 𝒂𝒅
𝒂𝟏⊤𝒙⋮
𝒂𝒅⊤𝒙
= 2𝐴⊤𝐴𝒙
!
Claim (next slide)
𝑖=1
𝑑
𝑐𝑖𝒚𝒊 = (𝒚𝟏…𝒚𝒅)𝒄
𝐴𝒙 =𝒂𝟏⊤
⋮𝒂𝒅⊤
𝒙 =
𝑎11𝑥1 +⋯+ 𝑎1𝑑𝑥𝑑⋮
𝑎𝑑1𝑥1 +⋯+ 𝑎𝑑𝑑𝑥𝑑
𝛻 𝐴𝒙 ⊤ 𝐴𝒙 = 𝛻
𝑖=1
𝑑
𝑎𝑖1𝑥1 +⋯+ 𝑎𝑖𝑑𝑥𝑑2 =
𝑖=1
𝑑
𝛻 𝑎𝑖1𝑥1 +⋯+ 𝑎𝑖𝑑𝑥𝑑2
linearity of 𝛻
Apex. Proof of Rem. 10-1 (contd.)29
Claim
𝑖=1
𝑑
𝑐𝑖𝒚𝒊 = (𝒚𝟏…𝒚𝒅)𝒄
r. h. s. =
𝑦11 ⋯ 𝑦𝑑1⋮ ⋱ ⋮𝑦1𝑑 ⋯ 𝑦𝑑𝑑
𝑐1⋮𝑐𝑑
=
𝑦11𝑐1 +⋯+ 𝑦1𝑑𝑐𝑑⋮
𝑦𝑑1𝑐1 +⋯+ 𝑦𝑑𝑑𝑐𝑑
= 𝑐1
𝑦11⋮𝑦1𝑑
+⋯+ 𝑐𝑑
𝑦𝑑1⋮
𝑦𝑑𝑑
=
𝑖=1
𝑑
𝑐𝑖𝒚𝒊 = (l. h. s. )
Nonlinear Regression
Least Square Estimator31
AIC (Akaike Information criteria: 赤池情報量)
AIC = −2
𝑖=1
𝑛
log 𝑓 𝑋𝑖; 𝜃𝑛ML + 2dim(𝜃)
Nonlinear regression (非線形回帰)
Suppose
𝑦𝑖 = ℎ 𝑥𝑖 + 𝑒𝑖 and 𝑒𝑖 ∼ N(0, 𝜎2) where
ℎ 𝑥 = 𝛾0 + 𝛾1𝑥 + 𝛾2𝑥2 + 𝛾3𝑥
3 +⋯ .
Estimate ℎ such that
min𝛽
𝑖=1
𝑛
𝑦𝑖 − ℎ(𝑥𝑖)2
Top Related