The calculation for multi-variate regression

[softwareNerd]

This post has nothing to do with philosophy. Please skip it if you have only a layman's knowledge or interest in statistics.

I have spent a while Googling for a formula in the area of Multiple Regression (linear, least-square fit), that I want to use as a basis for a software algorithm. I thought I'd find it easily, but was surprised when I did not. Here's a description of what I'm looking for. (I am not using very formal notation in what follows, to keep things simple; and, I figure that someone who can answer will understand this notation.)

One familar formulation used to guess the value of Y(a dependent variable), from known values of x₁, x₂, x₃,...x_N (independent variables), is the following: Y = a + b₁.x₁ + b₂.x₂ + b₃.x₃ + ... b_N.x_N ('.' ... the dot symbolizing multiplication)

where b₁, b₂,b₃...b_N have been calculated/estimated from some sample data.

In the case where there is only a single independent variable (x1), we can estimate the values of b1 and a from the sample data as follows:

b₁ = Covariance(x,y) / Var(x)

Alternately,

b₁ = (n.S_xy - S_x.S_y) / (n.S_xx - S_xx) ... where S is being used to symbolize sigma/sum-of

and

a = Mean(Y) - b₁.Mean(x₁)

Now, if one has three independent variable (x₁, x₂ and x₃), one can use:

a = Mean(Y) - b₁.Mean(x₁) - b₂.Mean(x₂) - b₃.Mean(x₃)

My question is, what about the calculations to derive b₁, b₂ and b₃ ? At the stage where I need to calculate these three, I will already know all the Variances and Covariance... Var(x₁),Var(x₂), Var(x₃),Cov(x₁,x₂), Cov(x₁,x₃), Cov(x₂,x₃)...so, if I can speed things up by using these as input to the algorithm, that'll be good.

I've searched pages and pages of stuff on the Internet and even check a few textbooks. Problem is that many of them explain the conceptual ideas, assume that the actual calculations will be performed with a tool like the R-project tool or MATLAB. The best I got was a way to calculate with two independent variables, and I'm not sure how to extrapolate that to the three variable situation.

A web-link to a place where I can read up on this would be helpful. Alternatively, a web-link to a forum where I might find people who could give me a link would work too!

[necrovore]

I don't think I can work through the math tonight, but I noticed that Wolfram MathWorld has a page with the derivation for LeastSquaresFitting, and even though it is for only one variable, you could probably extend it to any number of variables. Actually it looks fairly simple.

You would end up with a system of equations and you would have to solve it with Gauss-Jordan Elimination.

[Edited to eliminate redundancy]

Edited January 11, 2007 by necrovore

[softwareNerd]

I did find one open-source Java program which was calculating a matrix of beta (b₁, b₂, b₃...), using two input matrices: X and Y that hold the sample data,

X looks like this (or like the transpose of this...not sure which):

x₁₁, x₁₂, x₁₃, x₁₄ ...

x₂₁, x₂₂, x₂₃,...

x₃₁, x₃₂,...

Y looks like (y₁, y₂, y₃, y₄...)

I don't know if the guy who coded this did it right, but here is what he does with the matrices. (This is the calculation where the interacept A is assumed to be zero):

1. Calculate TX <-- (transpose of X)
   
2. Calculate M1 <-- TX mult X (matrix multiplication)
   
3. Calculate M2 <-- Inverse of M1
   
4. Calculate M3 <-- M2 mult TX
   
5. Calculate B <-- M3 mult Y
   

And B, supposedly, will be the matrix of values being sought (b₁, b₂, b₃...).

I have to get this running with something small, to see if it looks right.

[DarkWaters]

I did find one open-source Java program which was calculating a matrix of beta (b₁, b₂, b₃...), using two input matrices: X and Y that hold the sample data,

X looks like this (or like the transpose of this...not sure which):

x₁₁, x₁₂, x₁₃, x₁₄ ...

x₂₁, x₂₂, x₂₃,...

x₃₁, x₃₂,...

Y looks like (y₁, y₂, y₃, y₄...)

I don't know if the guy who coded this did it right, but here is what he does with the matrices. (This is the calculation where the interacept A is assumed to be zero):

1. Calculate TX <-- (transpose of X)
   
2. Calculate M1 <-- TX mult X (matrix multiplication)
   
3. Calculate M2 <-- Inverse of M1
   
4. Calculate M3 <-- M2 mult TX
   
5. Calculate B <-- M3 mult Y
   

And B, supposedly, will be the matrix of values being sought (b₁, b₂, b₃...).

I have to get this running with something small, to see if it looks right.

Hi,

This is correct.

Given we want the least squares solution to a system of equations Bx = b (here the x's are our variables and the B and b matricies contain our data), the optimal solution is x = ( B^t B ) ^ -1 B^t b. This is precisely what the pseudocode you have provided does.

Good luck!

--DW

Edit: to remove emoticon with shades from mathematical formula.

Edited January 11, 2007 by DarkWaters

[softwareNerd]

Thanks to both. Since I'm not comfortable doing the math derivations myself, I simply want to implement (i.e. program) it by translating a reliable formula into Java. The Matrix program I found has some programming problems, but since you've confirmed it's got the right approach, I'll take a closer look at it. Meanwhile, I also plan on looking at The R-Project, to see if I can use some components from there.

I have one further question. If I already have a matrix with all the covariances calculated, can I use that in some way. i.e. is there a formula that uses that as one component? Basically, I already have the covariances cov(x_i,y) and also all the cov(x_i,x_j).

[Adrian Hester]

I have one further question. If I already have a matrix with all the covariances calculated, can I use that in some way. i.e. is there a formula that uses that as one component? Basically, I already have the covariances cov(x_i,y) and also all the cov(x_i,x_j).

I think this site has what you're looking for. Search for the section title "Obtaining b weights from a Correlation Matrix" about two-thirds of the way down. (I'm too tired right now to be sure I'm thinking of the right formula for the covariance, or cross-covariance, matrix, so I'll leave that to you.)

[softwareNerd]

Looks hopeful. I'll study it closer when I'm less sleepy. Thanks.