Introduction
This case study focuses on a project aimed at creating an empathy analysis model to generate empathy scores for conversations. The project’s objective was to develop a model that could assess the level of empathy in various conversation datasets, including hospitalist data, helpline conversations, and sales datasets. The case study covers the dataset gathering process, model training, optimization to reduce size while maintaining accuracy, and the use of ONNX for framework independence and improved inference time.
Project Overview
Project Goals :
- Develop an empathy analysis model to generate empathy scores for conversations.
- Gather and merge diverse datasets, including hospitalist data, helpline conversations, and sales datasets, in JSON and CSV formats.
- Clean and annotate the datasets to create a final high-quality dataset for training and evaluation.
- Train the model using a BERT model with 12 layers.Optimize the model’s size while maintaining accuracy by adjusting the model architecture.
- Optimize the model’s size while maintaining accuracy by adjusting the model architecture.
Methodology :
- Dataset Gathering and Preprocessing : The project began by gathering various conversation datasets, including hospitalist data, helpline conversations, and sales datasets, in JSON and CSV formats. These datasets were merged, cleaned, and annotated to generate a final dataset suitable for training the empathy analysis model. Considerable effort was made to ensure data quality and include a diverse range of conversation types.
- Model Training : BERT model with 12 layers was utilized for training the model. The training process involved feeding the model with the annotated dataset to learn and capture patterns related to empathy in conversations.
- Model Optimization : The BERT model that was employed achieved an accuracy of 88% but had a large size of around 450 MB. To reduce the model’s size while maintaining accuracy, a DistilBERT model with 6 layers was utilized, resulting in a size of 270 MB while maintaining an accuracy of 88%. Further reduction in size was achieved by decreasing the model’s layers to 1, resulting in a size of 128 MB but decreasing accuracy to 73%.To address the decrease in accuracy, efforts were made to improve the quality and quantity of the dataset. Techniques such as data augmentation, additional data collection, and manual annotation were employed. With these enhancements, the model’s accuracy was improved to 90%, while successfully maintaining a size of 128 MB.To improve framework independence and decrease inference time, the optimized empathy analysis model was exported to ONNX (Open Neural Network Exchange). This also facilitated frontend developers’ inclusion of the model within their JavaScript-based applications.
Results and Deployment
The final empathy analysis model achieved an accuracy of 90% while maintaining a reduced size of 128 MB. The model demonstrated its ability to accurately assess empathy levels in various conversation datasets.
The deployment of the model involved hosting the APIs and model on AWS Lambda, utilizing AWS EFS for scalable file storage, and deploying the APIs on AWS EC2 instances. This setup ensured efficient scalability and accessibility for real-time empathy analysis.
Conclusion
This case study highlighted the successful development of an empathy analysis model for conversations, encompassing dataset gathering, preprocessing, model training, optimization, and deployment. The project’s outcomes contribute to improved understanding and assessment of empathy levels in conversations, enhancing communication and customer satisfaction.
