Algorithm-specific effects of class balancing: statistical evidence from Turkish institutional document classification

Abstract

Imbalanced class distributions pose significant challenges in document classification, but which balancing methods work best with different algorithms remains insufficiently understood. This research conducts rigorous statistical analysis of five balancing methods (SMOTE, ADASYN, BorderlineSMOTE, SMOTEENN, SMOTETomek) applied to twelve machine learning classifiers. We use a novel dataset of 2,240 Turkish institutional documents from the Social Security Institution with a marginal class imbalance (2.2:1 ratio) across five functional classifications - a regime frequently encountered in real institutional settings yet underexamined in the balancing literature, which has concentrated on severely imbalanced benchmarks. Through stratified 10-fold cross-validation with paired t-tests and Cohen's d effect-size measurements across 72 experimental configurations (yielding 60 pairwise statistical comparisons), we observed algorithm-dependent performance differences that question universal balancing assumptions. SMOTEENN reduced performance for ten of the twelve classifiers at the p < 0.05 level (mean Delta F1 = -0.0495, mean Cohen's d = -1.57; AdaBoost and Naive Bayes showed negative but non-significant changes), while alternative techniques produced algorithm-specific outcomes: ensemble methods showed only descriptive, non-statistically-significant point gains (Random Forest with SMOTE: Delta F1 = +0.0043, p = 0.470; AdaBoost with SMOTE: Delta F1 = +0.0225, p = 0.174), whereas the distance-based K-Nearest Neighbors classifier experienced uniform degradation across every balancing method (Delta F1 = -0.0613 to -0.0328, all p < 0.05, all large effects). Independent test validation confirmed results with minimal generalization differences (mean absolute Delta F1 = 0.034 across all 72 configurations). This research introduces the first Turkish institutional document classification dataset, provides statistically robust evidence questioning the universal effectiveness of balancing techniques, and suggests that, in the marginal imbalance regime observed in this institutional dataset, automatic balancing may not be necessary and may even degrade performance for certain algorithm families. Findings highlight the importance of algorithm-specific validation rather than one-size-fits-all approaches in resource-constrained settings.