Ridge Regression and Kernel Ridge Regression_椿树秋水

http://blog.sina.com.cn/u/3944981073

首页博文目录关于我

个人资料

微博

加好友发纸条

写留言加关注

博客等级：
博客积分：

博客访问：
关注人气：
获赠金笔：0支
赠出金笔：0支
荣誉徽章：

正文字体大小：大中小

Ridge Regression and Kernel Ridge Regression

(2017-06-25 17:22:39)

当用线性y=X*Beta回归去实现回归模型，而X.T*X不可逆时，简单的线性回归不可实现，这时可采用岭迹回归。它的思想是线性回归时无偏估计，但是用线性回归去实现时，方差不一定最小，也就是拟合不足，假如加上一个参数使得方差和无偏估计达到一定的平衡以避免拟合不足，实现风险最小。

一、岭迹回归的实现

（1）原理

X.T*X不可逆，此时可加上一个对角矩阵lamda*I使得inverse（ X.T*X+lamda*I）能实现，此时回归系数：

Beta=inverse(X.T*X+lamda*I)*(X.T*y)

实际上岭迹回归是L2正则化，它的严格推导为：

http://s11/mw690/004iYJMZzy7c9jDGA3w6a&690Regression and Kernel Ridge Regression" TITLE="Ridge Regression and Kernel Ridge Regression" />

实现上式最小化，对上式求导并令其为零，解得：

http://s8/mw690/004iYJMZzy7c9jJRjZd27&690Regression and Kernel Ridge Regression" TITLE="Ridge Regression and Kernel Ridge Regression" />

（2）python实现

from numpy import *

class selection(object):

def zscore(self,dataSet):

n,p=dataSet.shape

meanVal=mean(dataSet,axis=0)

meanMinus=dataSet-meanVal

#print meanVal

stdVal=std(dataSet,axis=0)

#print stdVal

zscore=meanMinus/(stdVal*sqrt(n))

return zscore

def rige(self,xdata,ydata,lamda):

matxdata=mat(xdata.astype(float))

n=shape(matxdata)[1]

matxdata=self.zscore(matxdata)

matydata=mat(ydata.astype(float))

ystd_data=mat(matydata)-mean(mat(matydata),axis=0)

beta=linalg.inv(matxdata.T*matxdata+eye(n)*lamda)*matxdata.T*ystd_data

predicty=array(matxdata*beta+mean(mat(matydata),axis=0))

return beta,predicty

def rssError(self,ydata,predictydata):

ydata=array(ydata.astype(float))

#ystd_data=mat(matydata)-mean(mat(matydata),axis=0)

return ((ydata-array(predictydata))**2).sum()

二、kernel Ridge实现

（1）原理

将数据的特征映射到更高维的空间，通过函数Xi->，这样我们可以通过数据样本的数量或者特征求逆。

beta=inverse(K+lamda*I)*y

其中k为通过核函数实现的矩阵。

http://s13/mw690/004iYJMZzy7c9lcms3O0c&690Regression and Kernel Ridge Regression" TITLE="Ridge Regression and Kernel Ridge Regression" />

http://s9/mw690/004iYJMZzy7c9le2OPSc8&690Regression and Kernel Ridge Regression" TITLE="Ridge Regression and Kernel Ridge Regression" />

（2）python实现

# coding:utf-8

from numpy import *

class selection(object):

def Gauss_kernel(self,x,z,sigma=10):

return exp(-sum((x-z)**2)/(2*sigma**2))

def zscore(self,dataSet):

n,p=dataSet.shape

meanVal=mean(dataSet,axis=0)

meanMinus=dataSet-meanVal

#print meanVal

stdVal=std(dataSet,axis=0)

#print stdVal

zscore=meanMinus/(stdVal*sqrt(n))

return zscore

def rige(self,xdata,ydata,lamda):

matxdata=mat(xdata.astype(float))

n=shape(matxdata)[1]

matxdata=self.zscore(matxdata)

matydata=mat(ydata.astype(float))

ystd_data=mat(matydata)-mean(mat(matydata),axis=0)

beta=linalg.inv(matxdata.T*matxdata+eye(n)*lamda)*matxdata.T*ystd_data

predicty=array(matxdata*beta+mean(mat(matydata),axis=0))

return beta,predicty

def rssError(self,ydata,predictydata):

ydata=array(ydata.astype(float))

#ystd_data=mat(matydata)-mean(mat(matydata),axis=0)

return ((ydata-array(predictydata))**2).sum() #transforming to array can do **2

def kernelRige(self,xdata,ydata,lamda,sigma,kernel='Gauss_kernel'):

matxdata=mat(xdata.astype(float))

m,n=matxdata.shape

matxdata=self.zscore(matxdata)

matydata=mat(ydata.astype(float))

ystd_data=mat(matydata)-mean(mat(matydata),axis=0)

xx=array(matxdata).transpose()

k=zeros((m,m))

for i in range(m):

for j in range(m):

k[i,j]=self.Gauss_kernel(xx[:,i],xx[:,j],sigma)

alpha=linalg.inv(k+eye(m)*lamda)*ystd_data

predicty=array(k*alpha+mean(mat(matydata),axis=0))

return alpha,predicty

三、实例

x	y
0.19035	0.878049
0.306657	-0.10941
0.017568	0.030917
0.122328	0.951109
0.076274	0.774632
0.614127	-0.25004
0.220722	0.807741
0.08943	0.840491
0.278817	0.34221
0.520287	-0.9503
0.726976	0.852224
0.180485	1.141859
0.801524	1.012061
0.474273	-1.31123
0.345116	-0.31991
0.981951	-0.3742
0.127349	1.039361
0.75712	1.040152
0.345419	-0.42976
0.314532	-0.07576
0.250828	0.657169
0.431255	-0.90544
0.386669	-0.50888
0.143794	0.844105
0.470839	-0.95176
0.093065	0.785034
0.205377	0.7154
0.083329	0.853025
0.243475	0.699252
0.062389	0.567589
0.764116	0.834931
0.018287	0.199875
0.973603	-0.35975
0.458826	-1.11318
0.5112	-1.08256
0.712587	0.615108
0.464745	-0.83575
0.984328	-0.3325
0.414291	-0.80882
0.799551	1.072052
0.499037	-0.9245
0.966757	-0.19164
0.756594	0.991844
0.444938	-0.96953
0.410167	-0.77343
0.532335	-0.63177
0.343909	-0.31331
0.854302	0.719307
0.846882	0.916509
0.740758	1.009525
0.150668	0.832433
0.177606	0.893017
0.445289	-0.89824
0.734653	0.787282
0.559488	-0.66348
0.232311	0.499122
0.934435	-0.12153
0.219089	0.823206
0.636525	0.053113
0.307605	0.0275
0.713198	0.693978
0.116343	1.242458
0.680737	0.36891
0.48473	-0.89194
0.929408	0.234913
0.008507	0.103505
0.872161	0.816191
0.75553	0.985723
0.620671	0.026417
0.47226	-0.96745
0.257488	0.6301
0.130654	1.025693
0.512333	-0.8843
0.74771	0.849468
0.669948	0.413745
0.644856	0.253455
0.894206	0.482933
0.820471	0.899981
0.790796	0.922645
0.010729	0.032106
0.846777	0.768675
0.349175	-0.32293
0.453662	-0.95771
0.624017	-0.16991
0.211074	0.86984
0.062555	0.60718
0.739709	0.859793
0.985896	-0.43363
0.782088	0.97638
0.642561	0.147023
0.779007	0.913765
0.185631	1.021408
0.52525	-0.70622
0.236802	0.564723
0.440958	-0.99378
0.39758	-0.70819
0.823146	0.860086
0.370173	-0.64923
0.791675	1.162927
0.456647	-0.95684
0.11335	0.850107
0.351074	-0.3061
0.182684	0.825728
0.914034	0.305636
0.751486	0.898875
0.216572	0.974637
0.013273	0.062439
0.469726	-1.22619
0.060676	0.599451
0.77631	0.902315
0.061648	0.464446
0.714077	0.947507
0.559264	-0.71511
0.121876	0.791703
0.330586	-0.16582
0.662909	0.379236
0.785142	0.96703
0.161352	0.979553
0.985215	-0.3177
0.457734	-0.89073
0.171574	0.963749
0.334277	-0.26623
0.501065	-0.91031
0.988736	-0.47622
0.659242	0.218365
0.359861	-0.33873
0.790434	0.843387
0.462458	-0.91165
0.823012	0.813427
0.594668	-0.60302
0.498207	-0.87885
0.574882	-0.4196
0.570048	-0.44209
0.33157	-0.34757
0.195407	0.822284
0.814327	0.974355
0.641925	0.073217
0.238778	0.657767
0.400138	-0.7156
0.670479	0.469662
0.069076	0.680958
0.294373	0.145767
0.025628	0.179822
0.697772	0.506253
0.729626	0.786519
0.293071	0.259997
0.531802	-1.09583
0.487338	-1.03448
0.21578	0.933506
0.625818	0.103845
0.179389	0.892237
0.192552	0.915516
0.671661	0.330361
0.952391	-0.06026
0.795133	0.945157
0.950494	-0.07186
0.194894	1.00086
0.35146	-0.22795
0.863456	0.648456
0.945221	-0.04567
0.77984	0.979954
0.996606	-0.4505
0.632184	-0.03651
0.790898	0.99489
0.022503	0.386394
0.318983	-0.15275
0.369633	-0.42396
0.1573	0.962858
0.153223	0.882873
0.360068	-0.65374
0.433917	-0.8725
0.133461	0.879002
0.757252	1.123667
0.309391	-0.10206
0.195586	0.925339
0.240259	0.689117
0.340591	-0.45504
0.243436	0.41576
0.612755	-0.18084
0.089407	0.723702
0.469695	-0.98786
0.94356	-0.0973
0.177241	0.918082
0.317756	-0.2229
0.515337	-0.73367
0.344773	-0.25689
0.537029	-0.79727
0.626878	0.048719
0.20894	0.836531
0.470697	-1.08028
0.054448	0.624676
0.10923	0.816921
0.158325	1.044485
0.97665	-0.30906
0.643441	0.267336
0.215841	1.018817
0.905337	0.409871
0.154354	0.920009
0.947922	-0.11238
0.201391	0.768894

sel=selection()

start=1;end=30;step=1;sigma=0.01

beta,predicty=featureSelection.kernelRige(x,y,start,sigma)

rssy=array(featureSelection.rssError(ydata,predicty))

#print rssy

for i in arange(start+step,end,step):

tempbeta,tempy=featureSelection.kernelRige(x,y,exp(i-10),sigma)

temprssy=featureSelection.rssError(y,tempy)

predicty=column_stack((predicty,tempy))

rssy=column_stack((rssy,temprssy))

beta=column_stack((beta,tempbeta))

minrss=min(rssy.ravel())

minindex=where(rssy.ravel()==minrss)

minlamda=arange(start,end,step)[minindex]

plt.plot(array(arange(start,end,step)),array(beta.T))

plt.show()

plt.scatter(x,y')

plt.scatter(x,predicty[i,minindex[0][0]],color='red')

plt.show()

结果如下图：

http://s14/mw690/004iYJMZzy7c9mNIbJX7d&690Regression and Kernel Ridge Regression" TITLE="Ridge Regression and Kernel Ridge Regression" />

http://s13/mw690/004iYJMZzy7c9mR47y40c&690Regression and Kernel Ridge Regression" TITLE="Ridge Regression and Kernel Ridge Regression" />

阅读┊ 收藏 ┊ 喜欢 ▼ ┊打印┊举报/Report

前一篇：2017年5月房价增长情况

后一篇：2017年6月房价情况

新浪BLOG意见反馈留言板　欢迎批评指正