Convex Structure Learning in Log Linear Models: Beyond Pairwise Potentials

Description

Convex Structure Learning in Log-Linear Models: Beyond Pairwise Potentials Mark Schmidt and Kevin Murphy Department of Computer Science University of British Columbia May 15, 2010

• beyond pairwise


• introduction higher-order


• gaussian graphical


• parameters wi


• structure learning

Sujets

• Schmidt
• Murphy

Informations

• Publié par: pefav
• Nombre de lectures: 49
• Poids de l'ouvrage: 3 Mo

Extrait

Convex Structure Learning in Log-Linear Models: Beyond Pairwise Potentials

Mark Schmidt and Kevin Murphy

Department of Computer Science University of British Columbia

May 15, 2010

Outline

1

2

3

4

5

Introduction Higher-Order Log-Linear Models Optimization Experiments Conclusion

Introduction Structure Learning Our Contribution

Higher-Order

Optimization

Experiments

Conclusion

with

Log-Linear

Structure Learning with` 1 -Regularization Our Contribution

` 1 -Regularization

Models

Mark Schmidt and Kevin Murphy

Convex Structure Learning in Log-Linear Models

Structure Learning with` 1 -Regularization Our Contribution

Introduction Higher-Order Log-Linear Models Optimization Experiments Conclusion Structure Learning with` 1 noazit-lariRegu

Several authors have recently examined parameter estimation in graphical models with` 1 -regularization. Regularization and structure learning in a convex framework. First works looked at Gaussian graphical models. Recent works considerlog-linear modelsof discrete data.

Mark Schmidt and Kevin Murphy

Convex Structure Learning in Log-Linear Models

Introduction Higher-Order Log-Linear Models Structure Learning with` 1 -Regularization ribution O E p x t p i e m ri iz m a e ti n o ts n Our Cont Conclusion Structure Learning with` 1 -onzatilariRegu

For example, assume we have apairwiseundirected graphical model, p(x), Z 1 Y φ i (x i ) Y φ ij (x i ,x j ), i j>i with node parametersw i and edge parametersw ij .

Assume thatw ij =0is equivalent to removing the edge (i,j).

We can usegroup` 1 -regularizationfor simultaneous parameter estimation and structure learning:

n m w in− X logp(x i |w) +λ X X ||w ij || 2 , i=1i j>i

Mark Schmidt and Kevin Murphy

Convex Structure Learning in Log-Linear Models

Introduction Higher-Order Log-Linear Models ` 1 -Regularization O E pt p i e m ri iz m at e i n o t n s S O t u r r uc C t o u n r t e ri L b e u a t r i n o i n ng with x Conclusion Structure Learning with` 1 guRerila-noazit

For example, assume we have apairwiseundirected graphical model, p(x), Z 1 Y φ i (x i ) Y φ ij (x i ,x j ), i j>i with node parametersw i and edge parametersw ij .

Assume thatw ij =0is equivalent to removing the edge (i,j).

We can usegroup` 1 -regularizationfor simultaneous parameter estimation and structure learning:

n m w in− X logp(x i |w) +λ X X ||w ij || 2 , i=1i j>i

Mark Schmidt and Kevin Murphy

Convex Structure Learning in Log-Linear Models

Introduction Higher-Order Log-Linear Models Op x t p i e m r i i z m at e i n o ts n S O t u r r uc C tu o r nt e ri L b e u a t r i n o i n ng with` 1 -Regularization E Conclusion Structure Learning with` 1 nioatizaruleg-R

For example, assume we have apairwiseundirected graphical model, p(x), Z 1 Y φ i (x i ) Y φ ij (x i ,x j ), i j>i with node parametersw i and edge parametersw ij .

Assume thatw ij =0is equivalent to removing the edge (i,j).

We can usegroup` 1 -regularizationfor simultaneous parameter estimation and structure learning:

n m w in− X logp(x i |w) +λ X X ||w ij || 2 , i=1i j>i

Mark Schmidt and Kevin Murphy

Convex Structure Learning in Log-Linear Models

` 1 -Regularization

Introduction Higher-OrderLog-Li O ne p a ti r m M iz o at d i e o l n s Structure Learning with Our Contribution Experiments Conclusion Structure Learning with` 1 -eRugaliraziton

A list of papers on this topic (incomplete):

[Li & Yang, 2004], [Li & Yang, 2005], [Banerjee et al., 2006], [Huang et al.,2006],[Leeetal.,2006],[Meinshausen&Bu¨hlmann,2006], [Wainwright et al., 2006], [Dahinden et al., 2007], [Schmidt et al., 2007], [Shimamura et al., 2007], [Yuan & Lin, 2007], [d’ Aspremont et al., 2008], [Banerjee et al., 2008], [Dahl et al., 2008], [Duchi et al., 2008], [Friedman et al., 2008], [Kolar & Xing, 2008], [Levina et al., 2008],

[Schmidtetal.,2008],[Fan&Feng,2009],[H¨oling&Tibshirani,2009], [Krishnamurphy & d’Aspremont, 2009], [Lu, 2009a], [Lu, 2009b], [Marlin et al., 2009a], [Marlin et al., 2009b], [Schmidt et al., 2009], [Schmidt & Murphy, 2009], [Schnitzspan et al., 2009], [Yuan, 2009], [Vidaurre et al., 2010].

Mark Schmidt and Kevin Murphy

Convex Structure Learning in Log-Linear Models

` 1 -Regularization

Introduction Higher-Order Log-Linear Models Optimization S O t u r r uc C tu o r nt e ri L b e u a t r i n o i n ng with Experiments Conclusion Structure Learning with` 1 oinularizat-Reg

Many of these papers have made thepairwiseassumption:

[Li & Yang, 2004], [Li & Yang, 2005], [Banerjee et al., 2006], [Huang et al.,2006],[Leeetal.,2006],[Meinshausen&B¨uhlmann,2006], [Wainwright et al., 2006], [Dahinden et al., 2007], [Schmidt et al., 2007], [Shimamura et al., 2007], [Yuan & Lin, 2007], [d’ Aspremont et al., 2008], [Banerjee et al., 2008], [Dahl et al., 2008], [Duchi et al., 2008], [Friedman et al., 2008], [Kolar & Xing, 2008], [Levina et al., 2008],

[Schmidtetal.,2008],[Fan&Feng,2009],[H¨oling&Tibshirani,2009], [Krishnamurphy & d’Aspremont, 2009], [Lu, 2009a], [Lu, 2009b], [Marlin et al., 2009a], [Marlin et al., 2009b], [Schmidt et al., 2009], [Schmidt & Murphy, 2009], [Schnitzspan et al., 2009], [Yuan, 2009], [Vidaurre et al., 2010].

Mark Schmidt and Kevin Murphy

Convex Structure Learning in Log-Linear Models

` 1 -Regularization

Introduction Higher-OrderLog-LinearMode o l n s g with Structure Learnin O E p x t p i e m ri iz m a e ti nts Our Contribution Conclusion Structure Learning with` 1 on-ugeRiralitaz

Many of these papers have made thepairwiseassumption:

[Li & Yang, 2004],[Li & Yang, 2005], [Banerjee et al., 2006], [Huang et al.,2006],[Leeetal.,2006],[Meinshausen&B¨uhlmann,2006], [Wainwright et al., 2006], [Dahinden et al., 2007], [Schmidt et al., 2007], [Shimamura et al., 2007], [Yuan & Lin, 2007], [d’ Aspremont et al., 2008], [Banerjee et al., 2008], [Dahl et al., 2008], [Duchi et al., 2008], [Friedman et al., 2008], [Kolar & Xing, 2008], [Levina et al., 2008],

[Schmidtetal.,2008],[Fan&Feng,2009],[H¨oling&Tibshirani,2009], [Krishnamurphy & d’Aspremont, 2009], [Lu, 2009a], [Lu, 2009b], [Marlin et al., 2009a], [Marlin et al., 2009b], [Schmidt et al., 2009], [Schmidt & Murphy, 2009], [Schnitzspan et al., 2009], [Yuan, 2009], [Vidaurre et al., 2010].

Mark Schmidt and Kevin Murphy

Convex Structure Learning in Log-Linear Models

` 1 -Regularization

Introduction Higher-Order Log-Linear Models Optimization S O t u r r uc C tu o r n e tri L b e u a t r i n o i n ng with Experiments Conclusion Structure Learning with` 1 tazinoigeR-ralu

Many of these papers have made thepairwiseassumption:

[Li & Yang, 2004],[Li & Yang, 2005], [Banerjee et al., 2006], [Huang et al., 2006],[Lee et al., 2006][,nieMuahs&neslmanB¨uh06],n,20 [Wainwright et al., 2006]et al., 2007], [Schmidt et al., 2007],, [Dahinden [Shimamura et al., 2007], [Yuan & Lin, 2007], [d’ Aspremont et al., 2008], [Banerjee et al., 2008], [Dahl et al., 2008], [Duchi et al., 2008], [Friedman et al., 2008], [Kolar & Xing, 2008], [Levina et al., 2008],

[Schmidtetal.,2008],[Fan&Feng,2009],[H¨oling&Tibshirani,2009], [Krishnamurphy & d’Aspremont, 2009], [Lu, 2009a], [Lu, 2009b], [Marlin et al., 2009a], [Marlin et al., 2009b], [Schmidt et al., 2009], [Schmidt & Murphy, 2009], [Schnitzspan et al., 2009], [Yuan, 2009], [Vidaurre et al., 2010].

Mark Schmidt and Kevin Murphy

Convex Structure Learning in Log-Linear Models