Skip to content

Improved accuracy #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ankit-vaghela30 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Improved accuracy #35

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions hydrus/naive_bayes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,19 +44,31 @@ def fit(self, x, y):
y = y.collectAsMap() # {id: label}
y = self.ctx.broadcast(y)

# create from ((id, feature), value) RDD, a new RDD of dimension ((label, feature), value)
# View the features both by doc id and by label.
def doc_to_label(x):
((doc_id, feature), value) = x
label = y.value[doc_id]
return ((label, feature), value)
by_label = x.map(doc_to_label) # ((label, feature), value)
by_label = by_label.reduceByKey(lambda x, y: x+y)
by_label_map = by_label.collectAsMap()

@cbarrick cbarrick Feb 2, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is gonna be 4*vocab... yikes

@ankit-vaghela30 ankit-vaghela30 Feb 2, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know! But this is the second change which improved my accuracy.

If you remember the equation of likelyhood probability, in the nominator, we have to count the words in the class given the document. I was simply counting # of words in class.

I am using the by_label_map to fetch count of words given document and if the combination of (label, feature) does not exist, I am returning 0 as a count and of course for each time I get the word count, I am adding 1 to it. Our likelyhood nominator becomes (for each class, each word):

by_label_map.get((label, feature), 0) + 1

Important thing here is that, before this change I was not counting those (0+1)s !

I know the collectAsMap() looks ugly. It was a desperate attempt to execute the idea as soon as possible!


# we will cartesian class labels with RDD
cartesian_label = labels.cartesian(x) # (label, (id, feature), value)
#print('cartesian product is: ', cartesian_label.collect())

# removing the id,
def restructure_cartesian_product(a):
(label, ((id, feature), value)) = a
return ((label, feature), value)
cartesian_product_rdd = cartesian_label.map(restructure_cartesian_product)
#
# We calculate likelyhodd probability for word given class and take log of that
def calculate_likelyhood_probability(by_label):
((label, feature), value) = by_label
value = (counts[label]+1)/prob_denom.value[label]
value = (by_label_map.get((label, feature),0)+1)/prob_denom.value[label]
return ((label, feature), np.log(value))
log_likelyhood_probability = by_label.map(calculate_likelyhood_probability) # ((label, feature), log likelyhood value)
log_likelyhood_probability = cartesian_product_rdd.map(calculate_likelyhood_probability) # ((label, feature), log likelyhood value)

# For naive bayes, we need the list of labels,
# their log priors, and log likelyhood probability.
Expand Down