Comparing Algorithms in Sentiment Analysis on DUKCAPIL App Reviews on Playstore Using Ensemble Learning Methods
DOI:
https://doi.org/10.70609/gtech.v9i1.6020Keywords:
Artificial Intelligence, Ensemble Learning, Extra Trees, Machine Learning, Sentiment AnalysisAbstract
The development of information and communication technology has significantly influenced public services, particularly through the adoption of mobile applications like DUKCAPIL, which simplifies access to population administration services. This study aims to analyze sentiment regarding the application by employing ensemble learning techniques and the SMOTE method to address data imbalance. The Extra Trees algorithm is compared against nine other algorithms, including Random Forest, Gradient Boosting, and LSTM. Extra Trees achieves the highest accuracy of 95.29% and outperforms in precision, recall, and F1-score. Deep learning models showed improved accuracy from 76.34% in the initial epoch to 91.56% in the final epoch. XGBoost and Random Forest also demonstrated strong performances, with accuracies of 90.55% and 92.66%, respectively. The results underline the superiority of Extra Trees in terms of stability and accuracy while highlighting the potential of deep learning for model enhancement. These findings provide valuable insights for the development of mobile application-based public services.
References
Abedin, M. Z., Hajek, P., Sharif, T., Satu, M. S., & Khan, M. I. (2023). Modelling bank customer behaviour using feature engineering and classification techniques. Research in International Business and Finance, 65. https://doi.org/10.1016/j.ribaf.2023.101913
Alsayat, A. (2022). Improving Sentiment Analysis for Social Media Applications Using an Ensemble Deep Learning Language Model. Arabian Journal for Science and Engineering, 47(2), 2499–2511. https://doi.org/10.1007/s13369-021-06227-w
Arafa, A., El-Fishawy, N., Badawy, M., & Radad, M. (2022). RN-SMOTE: Reduced Noise SMOTE based on DBSCAN for enhancing imbalanced data classification. Journal of King Saud University - Computer and Information Sciences, 34(8), 5059–5074. https://doi.org/10.1016/j.jksuci.2022.06.005
Barongo, R. I., & Tibangayuka Mbelwa, J. (2024). Using machine learning for detecting liquidity risk in banks. Machine Learning with Applications, 15, 100511. https://doi.org/10.24433/CO.5061127.v1
Bitetto, A., Cerchiello, P., Filomeni, S., Tanda, A., & Tarantino, B. (2023). Machine learning and credit risk: Empirical evidence from small- and mid-sized businesses. Socio-Economic Planning Sciences, 90, 101746. https://doi.org/10.1016/j.seps.2023.101746
Chachoui, Y., Azizi, N., Hotte, R., & Bensebaa, T. (2024). Enhancing algorithmic assessment in education: Equi-fused-data-based SMOTE for balanced learning. Computers and Education: Artificial Intelligence, 6. https://doi.org/10.1016/j.caeai.2024.100222
Citterio, A., & King, T. (2023). The role of Environmental, Social, and Governance (ESG) in predicting bank financial distress. Finance Research Letters, 51, 103411. https://doi.org/10.1016/j.frl.2022.103411
Dantas, R. M., Firdaus, R., Jaleel, F., Neves Mata, P., Mata, M. N., & Li, G. (2022). Systemic Acquired Critique of Credit Card Deception Exposure through Machine Learning. Journal of Open Innovation: Technology, Market, and Complexity, 8(4). https://doi.org/10.3390/joitmc8040192
Diekson, Z. A., Prakoso, M. R. B., Putra, M. S. Q., Syaputra, M. S. A. F., Achmad, S., & Sutoyo, R. (2022). Sentiment analysis for customer review: Case study of Traveloka. Procedia Computer Science, 216, 682–690. https://doi.org/10.1016/j.procs.2022.12.184
Domashova, J., & Kripak, E. (2022). Development of a generalized algorithm for identifying atypical bank transactions using machine learning methods. Procedia Computer Science, 213(C), 101–109. https://doi.org/10.1016/j.procs.2022.11.044
Fitri, V. A., Andreswari, R., & Hasibuan, M. A. (2019). Sentiment analysis of social media Twitter with case of Anti-LGBT campaign in Indonesia using Naïve Bayes, decision tree, and random forest algorithm. Procedia Computer Science, 161, 765–772. https://doi.org/10.1016/j.procs.2019.11.181
Gadgil, K., Gill, S. S., & Abdelmoniem, A. M. (2023). A Meta-learning based Stacked Regression Approach for Customer Lifetime Value Prediction. Journal of Economy and Technology. https://doi.org/10.1016/j.ject.2023.09.001
Ganaie, M. A., Hu, M., Malik, A. K., Tanveer, M., & Suganthan, P. N. (2021). Ensemble deep learning: A review. https://doi.org/10.1016/j.engappai.2022.105151
Guerra, P., Castelli, M., & Côrte-Real, N. (2022). Machine learning for liquidity risk modelling: A supervisory perspective. Economic Analysis and Policy, 74, 175–187. https://doi.org/10.1016/j.eap.2022.02.001
Gupta, P., Varshney, A., Khan, M. R., Ahmed, R., Shuaib, M., & Alam, S. (2022). Unbalanced Credit Card Fraud Detection Data: A Machine Learning-Oriented Comparative Study of Balancing Techniques. Procedia Computer Science, 218, 2575–2584. https://doi.org/10.1016/j.procs.2023.01.231
Huang, H., Zavareh, A. A., & Mustafa, M. B. (2023). Sentiment Analysis in E-Commerce Platforms: A Review of Current Techniques and Future Directions. In IEEE Access (Vol. 11, pp. 90367–90382). Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/ACCESS.2023.3307308
Huang, S., Wang, Y., Wong, E. Y. C., & Yu, L. (2024). Ensemble learning with soft-prompted pretrained language models for fact checking. Natural Language Processing Journal, 7, 100067. https://doi.org/10.1016/j.nlp.2024.100067
Imakura, A., Kihira, M., Okada, Y., & Sakurai, T. (2023). Another use of SMOTE for interpretable data collaboration analysis. Expert Systems with Applications, 228. https://doi.org/10.1016/j.eswa.2023.120385
Iqbal, J., Saeed, A., & Khan, R. A. (2023). The relative importance of textual indexes in predicting the future performance of banks: A connection weight approach. Borsa Istanbul Review, 23(1), 240–253. https://doi.org/10.1016/j.bir.2022.10.004
Islam, M. A., Uddin, M. A., Aryal, S., & Stea, G. (2023). An ensemble learning approach for anomaly detection in credit card data with imbalanced and overlapped classes. Journal of Information Security and Applications, 78. https://doi.org/10.1016/j.jisa.2023.103618
Jafari, S., & Byun, Y. C. (2024). Efficient state of charge estimation in electric vehicles batteries based on the extra tree regressor: A data-driven approach. Heliyon, 10(4). https://doi.org/10.1016/j.heliyon.2024.e25949
Jim, J. R., Talukder, M. A. R., Malakar, P., Kabir, M. M., Nur, K., & Mridha, M. F. (2024). Recent advancements and challenges of NLP-based sentiment analysis: A state-of-the-art review. Natural Language Processing Journal, 6, 100059. https://doi.org/10.1016/j.nlp.2024.100059
Karbasi, M., Ali, M., Bateni, S. M., Jun, C., Jamei, M., & Yaseen, Z. M. (2024). Boruta extra tree-bidirectional long short-term memory model development for Pan evaporation forecasting: Investigation of arid climate condition. Alexandria Engineering Journal, 86, 425–442. https://doi.org/10.1016/j.aej.2023.11.061
Kelebercová, L., & Zozuk, N. Č. (2023). Sentiment classification of annual reports in Slovak language using symbolic, subsymbolic and statistic methods. Procedia Computer Science, 225, 3508–3516. https://doi.org/10.1016/j.procs.2023.10.346
Koyyada, S. P., & Singh, T. P. (2022). A multi stage approach to handle class imbalance: An ensemble method. Procedia Computer Science, 218, 2666–2674. https://doi.org/10.1016/j.procs.2023.01.239
KP, V., AB, R., HL, G., Ravi, V., & Krichen, M. (2024). A tweet sentiment classification approach using an ensemble classifier. International Journal of Cognitive Computing in Engineering, 5, 170–177. https://doi.org/10.1016/j.ijcce.2024.04.001
Kristóf, T., & Virág, M. (2022). EU-27 bank failure prediction with C5.0 decision trees and deep learning neural networks. Research in International Business and Finance, 61. https://doi.org/10.1016/j.ribaf.2022.101644
Liu, Z., Zhang, Z., Yang, H., Wang, G., & Xu, Z. (2023). An innovative model fusion algorithm to improve the recall rate of peer-to-peer lending default customers. Intelligent Systems with Applications, 20. https://doi.org/10.1016/j.iswa.2023.200272
Mao, Y., Liu, Q., & Zhang, Y. (2024). Sentiment analysis methods, applications, and challenges: A systematic literature review. In Journal of King Saud University - Computer and Information Sciences (Vol. 36, Issue 4). King Saud bin Abdulaziz University. https://doi.org/10.1016/j.jksuci.2024.102048
Matrane, Y., Benabbou, F., & Sael, N. (2023). A systematic literature review of Arabic dialect sentiment analysis. In Journal of King Saud University - Computer and Information Sciences (Vol. 35, Issue 6). King Saud bin Abdulaziz University. https://doi.org/10.1016/j.jksuci.2023.101570
Mian, Z., Deng, X., Dong, X., Tian, Y., Cao, T., Chen, K., & Jaber, T. Al. (2024). A literature review of fault diagnosis based on ensemble learning. Engineering Applications of Artificial Intelligence, 127. https://doi.org/10.1016/j.engappai.2023.107357
Mienye, I. D., & Sun, Y. (2022). A Survey of Ensemble Learning: Concepts, Algorithms, Applications, and Prospects. In IEEE Access (Vol. 10, pp. 99129–99149). Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/ACCESS.2022.3207287
Mohammed, A., & Kora, R. (2022). An effective ensemble deep learning framework for text classification. Journal of King Saud University - Computer and Information Sciences, 34(10), 8825–8837. https://doi.org/10.1016/j.jksuci.2021.11.001
Okoro, E. E., Obomanu, T., Sanni, S. E., Olatunji, D. I., & Igbinedion, P. (2022). Application of artificial intelligence in predicting the dynamics of bottom hole pressure for under-balanced drilling: Extra tree compared with feed forward neural network model. Petroleum, 8(2), 227–236. https://doi.org/10.1016/j.petlm.2021.03.001
Pattnaik, D., Ray, S., & Raman, R. (2024). Applications of artificial intelligence and machine learning in the financial services industry: A bibliometric review. Heliyon, 10(1). https://doi.org/10.1016/j.heliyon.2023.e23492
Priyambada, S. A., Usagawa, T., & ER, M. (2023). Two-layer ensemble prediction of students’ performance using learning behavior and domain knowledge. Computers and Education: Artificial Intelligence, 5. https://doi.org/10.1016/j.caeai.2023.100149
Schmitt, M. (2023). Automated machine learning: AI-driven decision making in business analytics. Intelligent Systems with Applications, 18. https://doi.org/10.1016/j.iswa.2023.200188
Sharma, D., Kumar, R., & Jain, A. (2022). Breast cancer prediction based on neural networks and extra tree classifier using feature ensemble learning. Measurement: Sensors, 24. https://doi.org/10.1016/j.measen.2022.100560
SUDHAMATHI, T., & Perumal, K. (2024). ENSEMBLE REGRESSION BASED EXTRA TREE REGRESSORFOR HYBRID CROP YIELD PREDICTION SYSTEM. Measurement: Sensors, 101277. https://doi.org/10.1016/j.measen.2024.101277
Sun, Z., Harit, A., Cristea, A. I., Wang, J., & Lio, P. (2023). MONEY: Ensemble learning for stock price movement prediction via a convolutional network with adversarial hypergraph model. AI Open, 4, 165–174. https://doi.org/10.1016/j.aiopen.2023.10.002
Tiwari, D., Nagpal, B., Bhati, B. S., Mishra, A., & Kumar, M. (2023). A systematic review of social network sentiment analysis with comparative study of ensemble-based techniques. Artificial Intelligence Review, 56(11), 13407–13461. https://doi.org/10.1007/s10462-023-10472-w
Uddin, N., Uddin Ahamed, M. K., Uddin, M. A., Islam, M. M., Talukder, M. A., & Aryal, S. (2023). An ensemble machine learning based bank loan approval predictions system with a smart application. International Journal of Cognitive Computing in Engineering, 4, 327–339. https://doi.org/10.1016/j.ijcce.2023.09.001
Umer, M., Sadiq, S., karamti, H., Abdulmajid Eshmawi, A., Nappi, M., Usman Sana, M., & Ashraf, I. (2022). ETCNN: Extra Tree and Convolutional Neural Network-based Ensemble Model for COVID-19 Tweets Sentiment Classification: ETCNN: COVID-19 Tweets Sentiment Classification. Pattern Recognition Letters, 164, 224–231. https://doi.org/10.1016/j.patrec.2022.11.012
Wang, M., & Ji, H. (2022). Research on a Risk Control Model Based on Selective Ensemble Algorithm of Hierarchical Clustering and Simulated Annealing. Procedia Computer Science, 202, 164–171. https://doi.org/10.1016/j.procs.2022.04.023
Wang, Y., Zhang, Y., Liang, M., Yuan, R., Feng, J., & Wu, J. (2023). National student loans default risk prediction: A heterogeneous ensemble learning approach and the SHAP method. Computers and Education: Artificial Intelligence, 5. https://doi.org/10.1016/j.caeai.2023.100166
Widodo, A. O., Setiawan, B., & Indraswari, R. (2024). Machine Learning-Based Intrusion Detection on Multi-Class Imbalanced Dataset Using SMOTE. Procedia Computer Science, 234, 578–583. https://doi.org/10.1016/j.procs.2024.03.042
Xiao, P., Liu, Z., Zhao, G., & Pan, P. (2024). Novel stacking models based on SMOTE for the prediction of rockburst grades at four deep gold mines. Underground Space. https://doi.org/10.1016/j.undsp.2024.03.004
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Putu Putrayasa, Ema Utami, Robert Marco
This work is licensed under a Creative Commons Attribution 4.0 International License.
