Master the Art of Training with DAI Starter Course

Table of Contents

1. Introduction
2. Experiment's Accuracy
3. Experiment's Time
4. Experiment's Interpretability
5. Training Settings Knobs
   • Accuracy Knob
   • Time Knob
   • Interpretability Knob
6. Feature Evolution
7. Final Pipeline
8. Experiment Preview
9. Changing Knob Values
10. Scorer Knob
11. Conclusion

Article

Introduction

When using H2O Driverless AI for experiments, one of the unique features available is the training settings. These settings provide initial setup options for the experiment's accuracy, time, interpretability, and evaluation metric, known as the scorer. In this article, we will explore these training settings and their impact on the experiment.

Experiment's Accuracy

The accuracy knob in the training settings allows users to adjust the level of accuracy desired for the experiment. Increasing the value on the accuracy knob means that H2O Driverless AI will perform more work, considering various factors during the feature engineering, training, and selection process. This affects the feature evolution and the final pipeline of the model. On the other HAND, decreasing the value prioritizes speed over accuracy.

Experiment's Time

The time knob determines the relative time required to complete the experiment. Higher settings mean that H2O Driverless AI has more time to perform extensive work and find the best model. However, this also means longer runtime. The experiment may be stopped early if it doesn't Show improvements for a specified number of iterations.

Experiment's Interpretability

The interpretability knob controls the complexity of the models built by H2O Driverless AI. A higher value indicates a need for a model that is explainable, while a lower value allows H2O Driverless AI to be more creative in generating accurate models, even if some engineered features may not make Sense to human analysts. This knob affects the feature pre-pruning strategy, monotonicity constraints, and the feature engineering search space.

Training Settings Knobs

The training settings panel includes three adjustable knobs: accuracy, time, and interpretability. These knobs allow users to fine-tune the experiment Based on their requirements and preferences.

• Accuracy Knob: Adjusting the accuracy knob influences the extent of feature engineering, training, and selection process. Increasing the value prioritizes accuracy over speed, while decreasing the value prioritizes speed.
• Time Knob: The time knob determines the relative time required for the experiment. Higher settings allow H2O Driverless AI to perform more work, but also result in longer runtime.
• Interpretability Knob: Changing the interpretability knob impacts the complexity of the models built. A higher value ensures a more easily explainable model, while a lower value allows H2O Driverless AI to be more creative in generating accurate models.

Feature Evolution

Feature evolution is a crucial aspect of the experiment's workflow. H2O Driverless AI uses a genetic algorithm to find the best set of model parameters and feature transformations. This process helps in creating a final model with optimized features.

Final Pipeline

The final pipeline determines the number of models used and the validation method employed in the experiment. The final pipeline is a result of the feature evolution and selection process conducted by H2O Driverless AI.

Experiment Preview

On the right side of the screen, users can find an experiment preview based on the selected settings. As each knob's value is adjusted, the experiment preview pane automatically updates to show how the change impacts the specific experiment.

Changing Knob Values

By modifying the values of the knobs in the training settings, users can observe the changes reflected on the left side of the screen. Adjusting the time knob, for example, affects the estimated runtime and the early stopping behavior. Similarly, changing the accuracy and interpretability knob values leads to updates in the feature evolution and model Type.

Scorer Knob

The scorer knob in the training settings determines the evaluation metric for the experiment. H2O Driverless AI chooses the best scorer based on the dataset. For classification problems, the suggested scorer is often the Area Under the Curve (AUC). However, users can manually select a different scorer by clicking on the scorer knob and exploring the available options.

Conclusion

In conclusion, the training settings in H2O Driverless AI play a significant role in shaping the experiments conducted. By adjusting the knobs for accuracy, time, and interpretability, users can fine-tune their experiments based on their requirements and priorities. The feature evolution, final pipeline, and scorer knob provide additional customization options to ensure the best possible outcome for the specific use case.

Highlights

• H2O Driverless AI provides unique training settings for experiments.
• The accuracy knob adjusts the trade-off between accuracy and speed.
• The time knob influences the experiment's runtime.
• The interpretability knob determines the complexity of the models.
• Feature evolution and final pipeline play crucial roles in model optimization.
• Experiment preview helps understand the impact of knob adjustments.
• The scorer knob selects the evaluation metric for the experiment.

FAQ

Q: How do the training settings affect the accuracy of the model? A: The accuracy knob in the training settings allows users to prioritize accuracy over speed. Increasing the value on this knob results in more extensive feature engineering, training, and selection processes, ultimately leading to a more accurate model.

Q: Does adjusting the time knob impact the quality of the model? A: Adjusting the time knob does not directly impact the quality of the model. Instead, it determines the relative time required to complete the experiment. Higher settings allow H2O Driverless AI to perform more work, which can potentially lead to finding a better model. However, this also increases the runtime of the experiment.

Q: How does the interpretability knob affect the models built by H2O Driverless AI? A: The interpretability knob controls the complexity of the models. A higher value ensures a more explainable model, making it easier to understand and interpret. On the other hand, a lower value allows H2O Driverless AI to be more creative, potentially generating more accurate models that may include engineered features that lack human interpretability.

Q: Can I manually select a different evaluation metric for the experiment? A: Yes, the scorer knob in the training settings allows users to manually select a different evaluation metric. H2O Driverless AI suggests the best scorer based on the dataset, but users have the option to choose a different one based on their specific needs and preferences.