Detecting hate speech on the Internet

Online bullying and trolls are the darker side of social media. Twitter's Chief Executive Officer stated in a February 2015 memo that he wanted to make it a corporate priority to detect abuse and hate speech on Twitter. And yet only a month ago, "Ghostbusters" star Leslie Jones was bombarded with racist tweets, a demonstration of how difficult it is to detect such abuse by automated means.

But there is good news. Yahoo has developed an algorithm that combined conventional keyword searches with more sophisticated machine learning techniques to detect more subtle examples of hate speech. They use word embeddings (word2vec) to convert a message into a vector, and then compare the vector of the message to known cases of hate speech. Their technique allowed them to reach an F-score of 0.783 and an AUC of 0.9055. Read their paper here or an article summarizing their paper here.