SARC



Step I

Training models used



  1. Binary - SARC: a model trained by examining the Self-Annotated Reddit Corpus to determine the sarcastic nature of text (read more here).
  2. Binary - Netflix Review Model: a model trained by examining film and TV show reviews on Netflix.
  3. Binary - Twitter: a model trained by examining tweets on Twitter.
  4. Categorical - Stanford Model: a model trained by examining data provided by Stanford University.


The SARC Model is a newly trained binary model that is used to classify text into sarcastic or not sarastic. More details on the dataset and how it was built can be found here and here. The dataset is rather complex and is comprised of multiple parts. For the purposes of a simpler description, only the most vital parts of the dataset will be discussed. First and foremost, the balanced training and testing datasets in the CSV format were parsed in a similar manner to extract the IDs of all the reponses to the Reddit comments and their respective labels (0 for not sarcastic and 1 for sarcastic). Having those IDs saved, the main JSON file containing all the metadata was parsed and searched for the response IDs saved previously. If the ID was found, the text of the reponse was extracted and labeled according to the previously saved response label. Since the dataset was previously divided into training and testing, both of them were simply shuffled and used for respective purposes. The labels were removed from the testing dataset and saved separately for the purposes of further testing. The dataset was thereafter split into seperate files (a format required by our parser) and used for testing the newly trained SARC model, as well as the three sentiment models as listed above.



The Twitter Model is a newly trained binary sentiment model. The dataset acquired from this website. The initial dataset contained 1,578,627 classified tweets labeled with 0 (negative) or 1 (positive). It was thereafter parsed to build a training dataset of the required format and shuffled to avoid potential bias. We then split the resulting dataset into 80:20 for training and testing purposes respectively.



To find out more about the models we trained please visit our Models and Datasets pages.





Dataset evaluated



  1. SARC - unlabeled (64,666): the balanced unlabeled dataset (~20%) created specifically for testing purposes.




Step II.A

SARC ANALYSIS - the SARC Ground Truth



  1. Correlation between the binary sentiment and the sarcastic nature (according to the Netflix model and SARC ground truth)
    2. SARCASTIC
    NOT SARCASTIC




  2. Correlation between the binary sentiment aand the sarcastic nature (according to the Twitter model and SARC ground truth)
    3. SARCASTIC
    NOT SARCASTIC




  3. Correlation between the categorical sentiment and the sarcastic nature (according to the Stanford model and SARC ground truth)
    4. SARCASTIC
    NOT SARCASTIC








Step II.B

SARC ANALYSIS - the SARC Model



  1. Correlation between the binary sentiment and the sarcastic nature (according to SARC + Netflix models)
    2. SARCASTIC
    NOT SARCASTIC




  2. Correlation between the binary sentiment aand the sarcastic nature (according to SARC + Twitter models)
    3. SARCASTIC
    NOT SARCASTIC




  3. Correlation between the categorical sentiment and the sarcastic nature (according to SARC + Stanford models)
    4. SARCASTIC
    NOT SARCASTIC








Step III

HT LEAD GENERATION - ANALYSING THE PERFORMANCE OF THE MODELS



  1. SARC ROC Curve


  2. Netflix ROC Curve


  3. Twitter ROC Curve