Integrating AI and Large Language Models for Automated Data Quality Enhancement in Data Integration Systems
Authors’ abstract
This article introduces an AI and LLM-based framework to automate data quality improvement in complex data systems. Traditional methods struggle with semantic inconsistencies and evolving schemas, degrading quality as data scales. The framework incorporates Real-Time Semantic Annotation (RTSA), adaptive ontology reinforcement, contextual similarity for duplicate detection, and continuous auto-healing. Explainability is ensured via SHAP-based alignment for transparency. Evaluated on the GOBY Benchmark dataset, it achieved 89.4% semantic annotation accuracy, outperforming the strongest baseline by 3%. The duplicate reduction rate was 64.5%, and the quality score averaged 83.2%, validating the auto-healing loop’s effectiveness. It adapts to evolving data without retraining, confirmed by robust performance under semantic drift. The explainability analysis showed a low SHAP divergence of 0.11, aligning model predictions with feature importance. An ablation study confirmed that the semantic feedback loop and ontology reinforcement significantly contribute to stability. This research offers a scalable solution for enhancing data quality by integrating AI-driven adaptability and LLM-based semantic understanding with built-in explainability for enterprise-grade data pipelines.
Paper
- Authors
- N.K. Ponon, M.A.R. Basani
- Publication date
- 2026
Transcript
A transcript has not been published for this episode yet.
Licensing
- Source license
- Creative Commons Attribution 4.0 International
- Recording license
- CC-BY-4.0
The authors and publisher do not sponsor or endorse this recording.