Source-Aware Evaluation of Tree-Based Intrusion Detection on MQTT-Associated IoT Devices

Abstract

Packet-level intrusion-detection benchmarks can reward capture-specific signals and conceal differences in class coverage. This study audits a tree-model benchmark on Gotham Dataset 2025, treating MQTT association as device scope. Reconstruction identifies 56 source files containing 8,629,880 packets and reproduces the archived 112,845-row experiment. Removing 2,585 unresolved Unknown rows leaves 110,260 packets for revised evaluation. Seven fixed configurations are compared over five random and five source-disjoint splits with explicit class-coverage requirements. Multiclass macro F1 is 0.7295 ± 0.1150 under random splitting and 0.6455 ± 0.0211 under source separation for the ensemble, compared with 0.8247 ± 0.0094 and 0.6617 ± 0.0207 for XGBoost-200. XGBoost-200 therefore outperforms the ensemble in mean multiclass F1 under both protocols. Source scores use a fixed 15-label denominator. The original holdout score used a different averaging set and cannot establish a directly comparable generalization gap. Feature exclusions, uncapped testing, chronological evaluation, and probability diagnostics reveal further limitations. The contribution is a reproducible evaluation protocol; the results establish neither an ensemble accuracy–efficiency advantage nor operational edge-device performance.