Evaluate Synthetic Data Quality with AutoML on Databricks

Table of Contents

1. Introduction
2. Synthetic Data Evaluation: Train-Synthetic-Test-Real Approach
3. Using the MOSTLY AI Platform and Databricks
4. Creating Datasets in Databricks
5. Three-Level Namespace in Databricks
6. Creating and Configuring Connectors in MOSTLY AI
7. Setting Up the Census Training Data SD Job
8. Running the Job and Checking the Output
9. Using Databricks ML Tools for Machine Learning
10. Creating AutoML Experiments in Databricks
11. Comparing AutoML Results for Original and Synthetic Data
12. Analyzing Model Metrics and Artifacts
13. testing the Models with Holdout Data
14. Registering and Using Models in Databricks
15. Batch Inference Using the Holdout Set
16. Comparing Predictions and Assessing Data Validity
17. Conclusion

Introduction

Synthetic data has become increasingly important in machine learning research and development as it offers a reliable way to evaluate the quality of data for downstream tasks. This article explores the process of evaluating synthetic data based on its utility using the Train-Synthetic-Test-Real evaluation approach. We will delve into the use of the MOSTLY AI platform and Databricks for generating and analyzing synthetic data. Additionally, we will discuss the creation of datasets in Databricks and the three-level namespace structure. The article will also cover the configuration of connectors in MOSTLY AI and the setup of the Census Training Data SD Job. Furthermore, we will explore the use of Databricks' ML tools for machine learning and creating AutoML experiments. The results from these experiments will be compared for both original and synthetic data. Additionally, we will explore various model metrics and artifacts. The article will also demonstrate how to test models using holdout data and register and use models in Databricks. Finally, we will analyze predictions and assess the validity of the synthetic data compared to the original data.

Synthetic Data Evaluation: Train-Synthetic-Test-Real Approach The Train-Synthetic-Test-Real approach is a robust method for evaluating the quality of synthetic data. This method acknowledges that machine learning models rely heavily on the accurate representation of deep underlying patterns to perform effectively on previously unseen data. While evaluating higher-level statistics can provide some insights, analyzing the utility of synthetic data for a specific task requires a more comprehensive approach. By following the Train-Synthetic-Test-Real evaluation approach, it is possible to achieve a more reliable assessment of synthetic data quality. The approach involves training machine learning models using a combination of real and synthetic data, and then testing the models using real, unseen data. The results obtained from testing against real data can provide valuable insights into the quality and accuracy of the synthetic data. This approach ensures that the synthetic data is representative of the underlying patterns and can effectively generalize to real-world scenarios.

Using the MOSTLY AI Platform and Databricks The MOSTLY AI platform and Databricks are two powerful tools that can be utilized for generating and analyzing synthetic data. MOSTLY AI provides a user-friendly interface for creating synthetic data, while Databricks offers extensive ML tools and infrastructure for running machine learning experiments. These platforms can be easily integrated, allowing for seamless data generation, testing, and analysis. By combining the capabilities of both platforms, data scientists can effectively evaluate the quality of synthetic data and gain valuable insights into its utility for downstream tasks.

Creating Datasets in Databricks Before diving into the evaluation process, it is necessary to create datasets in Databricks. Databricks operates using a three-level namespace structure, consisting of catalogs, databases, and tables. Data scientists can create catalogs and databases within catalogs to organize their data effectively. Within these databases, tables can be created to store the actual data. This structure allows for easy management and retrieval of data within Databricks. For the purpose of this evaluation, a dataset containing census data is used. The dataset is divided into a training set and a holdout set for testing purposes. By creating and organizing datasets in Databricks, data scientists can streamline the evaluation process and ensure the availability of relevant data for analysis.

Three-Level Namespace in Databricks The three-level namespace structure in Databricks provides a hierarchical organization of data, making it easier to manage and access datasets. At the top level, catalogs serve as containers for databases. Within catalogs, databases can be created to hold tables and other metadata. Finally, tables are used to store the actual data. This three-level structure offers a flexible and scalable approach to data organization in Databricks. By leveraging this structure, data scientists can efficiently manage and retrieve datasets, ensuring the availability of data for evaluation and analysis.

Creating and Configuring Connectors in MOSTLY AI To integrate MOSTLY AI with Databricks, it is necessary to establish connectors that enable seamless data transfer between the two platforms. MOSTLY AI allows users to create connectors specifically tailored to their needs. In the case of integrating with Databricks, two connectors should be created. The first connector, databricksDefault, should be configured to connect to the default database in Databricks, where the training and holdout datasets are stored. The second connector, databricksSynthOutput, should be configured to connect to a separate database within Databricks, where the synthetic data will be pushed once the job is completed. By establishing these connectors, data scientists can ensure a smooth flow of data between MOSTLY AI and Databricks, facilitating the evaluation and analysis of synthetic data.

Setting Up the Census Training Data SD Job In MOSTLY AI, the next step is to set up the Census Training Data SD Job. This job allows data scientists to generate synthetic data based on the training dataset in Databricks. Various settings can be configured within the job, depending on the specific requirements of the evaluation. For the purpose of this demo, the default settings are used, with Turbo enabled to expedite the generation process. Additionally, the destination for the generated synthetic data is set to the synth output table within Databricks. Once the job is run, it generates the synthetic data based on the underlying patterns in the original training dataset. This step is crucial as it produces the synthetic data for evaluation and analysis.

Running the Job and Checking the Output After setting up the Census Training Data SD Job, it can be run to generate the synthetic data. The job runs in MOSTLY AI, utilizing the underlying AI algorithms to create synthetic data that closely resembles the original dataset. Once the job completes successfully, the data scientist can check the output to ensure that the synthetic data has been generated as expected. In MOSTLY AI, the job log provides valuable information about the job run, including the number of rows generated, any QA reports, privacy metrics, and correlations. By analyzing this information, data scientists can assess the quality and accuracy of the synthetic data. It is important to ensure that the job runs without any errors and produces synthetic data that closely resembles the underlying patterns in the original dataset.

Using Databricks ML Tools for Machine Learning Databricks provides a suite of powerful ML tools that can be used for machine learning tasks. These tools offer out-of-the-box functionality, making it easy to train and evaluate machine learning models using the generated synthetic data. By leveraging these tools, data scientists can gain insights into the performance of models trained using the synthetic data. In the experiments tab of Databricks, the AutoML Experiment feature can be used to automate the process of training and evaluating models. The data scientist can create a cluster and configure various options such as the prediction target, evaluation metrics, and training frameworks. For this evaluation, the area under the curve evaluation metric is selected, and the experiment is run for both the original and synthetic data. By utilizing the ML tools in Databricks, data scientists can effectively train and evaluate machine learning models using the generated synthetic data.

Creating AutoML Experiments in Databricks To facilitate the training and evaluation of machine learning models, Databricks offers the AutoML Experiment feature. This feature allows data scientists to automate the entire process, from setting up the experiment to analyzing the results. By creating an AutoML Experiment, data scientists can easily compare the performance of different models on both the original and synthetic data. The experiment can be configured to use the generated datasets as inputs and select the appropriate prediction target. Advanced configurations, such as evaluation metrics and training frameworks, can also be customized to suit the specific requirements of the evaluation. By using AutoML Experiments in Databricks, data scientists can streamline the process of training and evaluating machine learning models on both original and synthetic data.

Comparing AutoML Results for Original and Synthetic Data After running the AutoML Experiment for both the original and synthetic data, data scientists can compare the results to gain insights into the performance of the models. Databricks logs various metrics and artifacts during the experiment, providing a comprehensive overview of the various models' performances. By analyzing these metrics, data scientists can assess the accuracy, precision, recall, and other important performance indicators of the models. In particular, the ROC-AUC score is a commonly used metric for binary classification problems. By comparing the results for both the original and synthetic data, data scientists can determine the efficacy of the synthetic data for the specific task at hand. Additionally, factors such as feature importance and model interpretability can also be considered to gain further insights into the models' behaviors.

Analyzing Model Metrics and Artifacts Databricks provides extensive logging of model metrics and artifacts during the AutoML Experiment. This logging ensures that data scientists have access to all the relevant information for analyzing and interpreting the results. By reviewing these metrics and artifacts, data scientists can gain a deeper understanding of the models' performances and behaviors. The logged artifacts include information such as the Python environment required to run the models, the model itself, and various HTML and PNG files that provide visual representations of the models and their outputs. By leveraging these artifacts, data scientists can effectively analyze and interpret the models' performances, making informed decisions about their effectiveness and utility.

Testing the Models with Holdout Data To validate the accuracy and reliability of the models trained on both the original and synthetic data, it is essential to test them with holdout data. The holdout data represents unseen data that was not used during the training process, ensuring an unbiased evaluation of the models. In Databricks, the models can be registered in a central repository, allowing easy access and utilization. By selecting the appropriate registered model, data scientists can perform batch inference on the holdout dataset. This process involves using the models to predict the target variable for the holdout data and comparing the predictions against the actual values. By analyzing the accuracy and reliability of the predictions, data scientists can assess the performance of the models trained on both the original and synthetic data.

Registering and Using Models in Databricks Databricks provides a central model registry that allows data scientists to register and manage their trained models. This registry serves as a centralized repository for storing and accessing models, providing convenient and efficient management of models at scale. By registering models in the model registry, data scientists can easily access and use them for various tasks, such as batch inference and real-time prediction. The models can be versioned, allowing for easy tracking and management of model versions. Additionally, Databricks offers built-in functionality for using registered models for inference, making it seamless to deploy and utilize the trained models in production environments.

Batch Inference Using the Holdout Set To assess the validity and reliability of the models, batch inference can be performed using the holdout dataset. In Databricks, batch inference involves using the registered models to predict the target variable for the holdout data. By comparing the predicted values against the actual values, data scientists can evaluate the accuracy and reliability of the models trained on both the original and synthetic data. The results of the batch inference can be analyzed to determine if the synthetic data can effectively generalize to unseen data and if it is comparable to the performance of the models trained on the original data. By performing batch inference, data scientists can gain valuable insights into the validity and utility of the synthetic data for the specific task at hand.

Comparing Predictions and Assessing Data Validity The predictions obtained from batch inference using the holdout dataset provide valuable insights into the validity and reliability of the synthetic data. By comparing the predictions from the models trained on both the original and synthetic data, data scientists can assess the accuracy and effectiveness of the synthetic data for the specific task. If the predictions from the synthetic data closely match those from the original data, it indicates that the synthetic data is of high quality and can be used reliably for downstream tasks. However, if there are significant differences in the predictions, it may raise concerns about the utility and validity of the synthetic data. By carefully analyzing and comparing the predictions, data scientists can make informed decisions about the suitability of the synthetic data for the specific task.

Conclusion

In conclusion, the Train-Synthetic-Test-Real approach provides a robust method for evaluating the quality of synthetic data. By utilizing platforms such as MOSTLY AI and Databricks, data scientists can generate and analyze synthetic data effectively. This evaluation process involves creating datasets in Databricks, establishing connectors in MOSTLY AI, setting up the Census Training Data SD Job, and running the job to generate the synthetic data. Databricks' ML tools can then be used to train and evaluate machine learning models on both the original and synthetic data. By comparing the results of the AutoML experiments, data scientists can gain insights into the efficacy of the synthetic data. Testing the models with holdout data allows for a reliable assessment of their accuracy and reliability. By registering and using the models in Databricks, data scientists can easily deploy and utilize the trained models in various scenarios. Ultimately, by comparing the predictions obtained from batch inference, data scientists can assess the validity and utility of the synthetic data for the specific task at HAND. Synthetic data provides a valuable tool for machine learning research and development, offering a reliable and privacy-preserving alternative to real data. With the right evaluation methodology and tools, data scientists can harness the power of synthetic data and unlock its potential for various downstream tasks.

Pros:

• Provides a comprehensive approach to evaluating synthetic data quality
• Combines the capabilities of MOSTLY AI and Databricks for seamless data generation and analysis
• Utilizes the three-level namespace structure in Databricks for effective data organization
• Offers the ability to create and configure connectors for efficient data transfer
• Allows for the setup and running of the Census Training Data SD Job in MOSTLY AI
• Utilizes Databricks' ML tools for easy training and evaluation of machine learning models
• Provides the AutoML Experiment feature in Databricks for automated model training and evaluation
• Enables comparison of AutoML results for original and synthetic data
• Logs various model metrics and artifacts for in-depth analysis
• Allows testing of models with holdout data for validation

Cons:

• Requires familiarity with the MOSTLY AI platform and Databricks tools
• The evaluation process can be time-consuming and resource-intensive
• The effectiveness of the synthetic data heavily depends on the underlying Patterns and the accuracy of the Generative AI algorithms

Highlights

• The Train-Synthetic-Test-Real approach offers a robust methodology for evaluating synthetic data quality.
• MOSTLY AI and Databricks are powerful platforms that can be integrated to generate and analyze synthetic data effectively.
• Databricks' ML tools provide out-of-the-box functionality for training and evaluating machine learning models on both original and synthetic data.
• The AutoML Experiment feature allows data scientists to automate the model training and evaluation process.
• Batch inference using holdout data enables the validation and reliability assessment of the models.
• Registering and using models in Databricks' model registry ensures easy access and deployment of trained models in production environments.

FAQs

Q: What is the Train-Synthetic-Test-Real approach? A: The Train-Synthetic-Test-Real approach is a methodology for evaluating synthetic data quality. It involves training machine learning models using a combination of real and synthetic data and then testing the models using real, unseen data. This approach provides a reliable assessment of the synthetic data's utility and its ability to generalize to real-world scenarios.

Q: How can MOSTLY AI and Databricks be integrated for synthetic data analysis? A: MOSTLY AI and Databricks can be integrated by establishing connectors between the two platforms. These connectors enable seamless data transfer, allowing the generated synthetic data to be analyzed using Databricks' ML tools. Integration between the platforms facilitates efficient evaluation of the synthetic data and enables data scientists to gain insights into its quality and utility.

Q: What is the three-level namespace structure in Databricks? A: The three-level namespace structure in Databricks provides a hierarchical organization of data. It consists of catalogs, databases, and tables. Catalogs serve as containers for databases, which in turn contain tables and other metadata. This structure allows for effective management and retrieval of data within Databricks, making it easier for data scientists to work with datasets.

Q: What is batch inference in Databricks? A: Batch inference in Databricks involves using trained models to predict outcomes for a set of data points. It is particularly useful for validating the accuracy and reliability of models trained on both original and synthetic data. By performing batch inference on holdout data, data scientists can assess the performance of the models and compare their predictions against the actual values.

Q: How does Databricks' model registry facilitate model management? A: Databricks' model registry serves as a centralized repository for storing and managing trained models. It allows data scientists to register their models, store different versions, and easily access and deploy them for various tasks. The model registry simplifies model management and ensures efficient deployment of models in production environments.