JoonHo Jang, Mask Modeling for Attention Mechanism in Sentiment Analysis, Master's Thesis, Department of Industrial and Systems Engineering, KAIST, 2019 

Abstract

In the case of models which use text data as input, such as Sentiment Analysis, Attention Mechanism is widely used to improve the performance by weighting the parts of input that affect the prediction. Despite the length constraints and the nature of the noisy text data, the existing Attention Mechanism models calculate the weights by using all the input words even if the length of sentence is extremely long. In this paper, we propose attention mask based on Bernoulli Sampling to remove noise among input words and to calculate the weights more precisely. Attention mask makes the Attention Mechanism use the selected hidden states when calculating weights. This paper also propose a way to process sentence by using CNN based residual learning instead of LSTM. In this model, experiments were conducted using Yelp data, which is widely used for Sentiment Analysis. From these experiments, we demonstrated that our proposed model have a better performance than baseline models.

@masterthesis{Jang:2019,

author = {JoonHo Jang},

advisor ={Il-Chul Moon},

title = {Mask Modeling for Attention Mechanism in Sentiment Analysis},

school = {KAIST},

year = {2019}

}