Written by Emtiyaz, CS, UBC.
Last updated: Dec. 17, 2011.
Table of Contents
This code can be used for the following categorical (binary + multiclass) data analysis,
 Binary/MultiClass classification (e.g. logistic regression).
 Binary/MultiClass Gaussian Process classification.
 Factor analysis/ PCA/ latent graphical model for categorical data.
 Missing value imputation of categorical data.
See some examples here examples of catLGM.
For full details of models/algorithms, see the following papers:
Download CatLGM.zip.
System requirements and dependencies: You need to download minFunc (by Mark Schmidt) required for numerical optimization and GPML toolbox for Gaussian processes.
Install these directories inside the catLGM directory.
I also use some functions from Tom Minka’s lightspeed toolbox for matrixinversion
and matrixdeterminant, but can be replaced by other equivalent functions.
catLGM code works fine on MATLAB 7.4 (2007a) and higher versions.
The following will get you started:
> addpath(genpath(pwd));
> exampleCatLGM;
All function calls follow the following format to perform a specific task on a data given a
model and an algorithm.
> out = catLGM(model, task, algo, data);
For example, given features X and labels y, the following performs posterior inference in
Bayesian logistic regression using a variational method based on the piecewise
bound, and then predict labels for new test data using Montecarlo.
> postDist = catLGM(’bayesLogitReg’, ’infer’, ’varpw’, [X y]);
> pred = catLGM(’bayesLogitReg’, ’predict’, ’mc’, Xtest, [], postDist);
Similarly, given a categorical data matrix Y with missing values, the following learns a Multinomiallogit factor
analysis model using a variational EM algorithm based on the log bound,
and then imputes the missing values using the learned model (with a
Montecarlo estimate).
> learnOut = catLGM(’multiLogitFA’, ’learn’, ’varlog’, Y);
> params = learnOut.params;
> postDist = learnOut.postDist;
> imputOut = catLGM(’multiLogitFA’, ’impute’, ’varlog’, Y, [], ’mc’, params, postDist);
Algorithm options can also be specified as 5’th argument.
> options = struct( 'maxItersInfer', 500, 'lowerBoundTolInfer', 0.001);
> postDist = catLGM(’bayesLogitReg’, ’infer’, ’varpw’, [X y], options);
See full list of options.
Following is a list of model, task and algo pairs in the current implementation
Logistic Regression Models The table below shows the models implemented.
Two tasks can be performed for these models: inference
(‘infer’) and prediction (‘pred’).
For task ‘infer’, algorithm options are variational EM
algorithms based on the Bohning bound (‘varboh’), the log bound
(‘varlog’), and the piecewise bounds (‘varpw’).
For ‘pred’ task, I use approximation based on
Montecarlo (‘mc’).
Model 
Description 
infer 
pred 
bayesLogitReg 
Bayesian logistic regression 
varboh varlog varpw 
mc 
bayesMultiLogitReg 
Bayesian Multinomiallogit regression 
varboh varlog 
mc 
See following papers for details of bounds:
Gaussian Process Classification
The table below shows the models implemented.
Currently, inference (‘infer’) and prediction (‘pred’) can be performed, but I will implement hyperparameter learning soon.
Algorithms are same as regression models.
See my AIstats paper for details of sticklikelihood and multiclass GP classification Categorical Data Analysis with Latent Gaussian Models.
Model 
Description 
infer 
pred 
logitGP 
Binary GP classification with logistic link 
varboh varlog varpw 
mc 
multiLogitGP 
Multiclass Gausian process classification with multinomiallogit link 
varboh varlog 
mc 
stickLogitGP 
Multiclass Gausian process classification with sticklogit link 
varpw 
mc 
Factor Models
Currently, learning (‘learn’), inference (‘infer’) and imputation (‘pred’) can be performed.
See my AIstats paper for details of the models and algorithms Categorical Data Analysis with Latent Gaussian Models.
Model 
Description 
learn and infer 
impute 
binaryLogitFA 
Binary Factor analysis with logistic link 
varboh varlog varpw 
mc 
binaryLogitLGGM 
Binary latent Gaussian graphical model with logistic link 
varboh varlog varpw 
mc 
multiLogitFA 
Categorical factor analysis with multinomiallogit link 
varboh varlog 
mc 
multiLogitLGGM 
Categorical latent Gaussian graphical model with multinomiallogit link 
varboh varlog 
mc 
stickLogitFA 
Categorical factor analysis with sticklogit link 
varpw 
mc 
stickLogitLGGM 
Categorical latent Gaussian graphical model with sticklogit link 
varpw 
mc 
Default values of options are set inside getCatLGMDefaultOptions.m
and getAlgoFuncsAndOptions.m.
Options for inference
 
maxItersInfer  Maximum number of iterations for inference step
100). 
lowerBoundTolInfer  Lower bound tolerance for inference step to
mine convergence (e.g. 0.01). 
initVec  Initialization for the inference step (a vector size of which
ds on the algorithm). 
displayInfer  Display on/off (1 or 0). 
Options for learning algorithm (EM)
 
maxItersLearn  Maximum number of E and M iterations.
learning 
lowerBoundTolLearn  Lower bound tolerance for inference step to
mine convergence (e.g. 0.01). 
displayLearn  Display on/off (1 or 0). 
computeSs  Compute sufficient statistics (switched
off during inference mode) 
maxItersMstep  Maximum number of iteration for Mstep (in case a gradient algorithm is used). 
lowerBoundTolMstep  Lower bound tolerance in Mstep (in case a gradient algorithm is used). 
displayMstep  Display on/off (1 or 0). 
Options for bound
 
boundParams  A structure containing bound parameters.

The hyperparameters can be passed through the options. If no hyperparameters
are specified, the default hyperparameter values are set inside setupLGM.m.
I will be adding the following in next few days. If you want something
specific, please email me.
 Add hyperparameter learning for multiclass GP.
 Binary GP methods such as EP from Carl Rasmussen’s GPML toolbox.
 Mark Girolami’s VB algorithm for MultiClass GP.
 Add the following bounds,
 The Jaakkola bound for logisticlogpartition function.
 Guillaume’s and Tom Minka’s bound for the logsumexp
function.
 Hybrid Monte Carlo (HMC) sampler for multiclass GP.
 Fast sparse GP and sparse latent graphical models.
