PROV4DistML: Provenance Data Management for the Analysis of Distributed Machine Learning Applications
PROV4DistML: Gerência de Dados de Proveniência para Análise de Aplicações de Aprendizado de Máquina Distribuído
Distributed Machine Learning (or simply Distributed Learning) and its branches, such as Federated Learning, represent approaches that enable collaboration among multiple devices or users in training Machine Learning models, including deep neural networks and clustering algorithms. These techniques have been widely applied in areas such as Astronomy, Medicine, and Biology due to their ability to accelerate the training of complex models by distributing the processing across multiple machines. In particular, Federated Learning allows training to occur without the need to centralize data, thereby preserving privacy and security, making it ideal for scenarios involving sensitive data, such as personal information. In a typical configuration, each worker (e.g., smartphones, computers, or clusters) trains a model locally and sends model updates to a central server. This server combines the contributions into a global model, which is redistributed for new training rounds. Distributed training is an iterative process that can be time-intensive, with its duration influenced by factors such as aggregation methods, hyperparameter choices, and the characteristics of the datasets used. The analysis of metrics such as accuracy and fine-tuning of configurations during distributed training enables important improvements, such as increased efficiency, better model interpretability, and higher fault tolerance. In this context, provenance data emerges as a promising solution to represent and track the lifecycle of artifacts generated during the distributed learning process, such as datasets, applied transformations, trained models, and adopted configurations. These data not only enable the monitoring and analysis of training but also allow interventions, such as dynamic parameter adjustments and configuration adaptations. The PROV4DistML project aims to develop solutions to capture, model, store, and manage provenance data in Distributed and Federated Learning applications. Additionally, the project seeks to leverage these data to support actions such as automatic parameter tuning, improving result interpretability, and increasing the robustness of training. In this way, PROV4DistML contributes to the development of more efficient, transparent, and resilient solutions in Distributed Learning.