Naive bayes

[sfsf9797]

• What I did

Implement Gaussian naive Bayes Class and Basic documentation.

• How I did it

Refer to the formula of Gaussian naive Bayes from the notes

• How to verify it

compare the performance with the gaussianNB from sci-kit learn.

[ddbourgin]

Amazing! At first glance this looks great, @sfsf9797 ! I'm pretty slammed with work right now, but am going to have a look at this this weekend.

[ddbourgin]

I think there are a few errors here.

• The log Gaussian likelihood calc in prob isn't quite right (see later commit for details)
• In the joint_log_likelihood = prior + prob line, you're adding a log-transformed vector (prob) to raw probabilities (prior), which doesn't make sense. I think what you want is np.log(prior) + prob. See my later commit for details.
• This isn't a joint likelihood, right? You're computing the the joint class likelihood in prob, but when you add in the prior, this will be proportional to the log class posterior.

[ddbourgin]

Nomenclature: I'm not sure it's right to call this the joint log likelihood? That is, this function computes the unnormalized quantity P(y = c | X, mean_c, sigma_c), which I'd think is the class posterior.

[ddbourgin]

@sfsf9797 this is what I believe the proper log likelihood + log class posterior calc should be. Let me know if you agree!

[ddbourgin]

Hey @sfsf9797 - Thanks for the PR! I just had a more thorough look and committed a few changes. Brief summary:

1. I moved this under the linear_models module rather than keeping it as a single hanging model.
2. I think there might have been a few bugs in the original log likelihood calc. I've committed what I believe is the correct version, but please go through it and make sure you agree.
3. I expanded the unit test you included -- comparing model accuracies is a good start (thanks!), but I think it actually masked some problems with the implementation. In particular, testing on multiple random cases revealed that there were mismatches in accuracy between sklearn and the current implementation, and comparing the actual class probabilities (rather than just the predictions) revealed a bug in the log posterior calculation.
4. I expanded the documentation to provide a better overview of the model.

Please feel free to make adjustments or ask questions. Once we both agree on the implementation and are happy with the model performance, I'm happy to merge.

[sfsf9797]

Hi, thank you so much for all the feedbacks, I will go through all these and get back to you this weekend.

[sfsf9797]

sorry, I am kind of busy these few weeks, but I will get you back latest by another 2 weeks times.

[sfsf9797]

yeah, true, naive Bayes is definitely under linear models when likelihood factors p(x∣c) are from exponential families.

[sfsf9797]

Hi @ddbourgin thanks for correcting my implementation. I have learnt a lot from you. I am pretty satisfied with the model after a few round of checking. Lastly, Thank you a lot for the all the comments!

[ddbourgin]

Awesome, merged! Thanks for the PR @sfsf9797 :)