The Rebirth of MariaDB: Building Bot Solutions

The Rebirth of MariaDB: Building Bot Solutions

Table of Contents

1. Introduction
2. Overview of the Continuous Integration framework
3. Keywords and Definitions
4. Schematic of the Buildbot framework
5. Multi-master setup for Buildbot
6. Docker Latent Workers
7. Advantages of using Docker Latent Workers
8. Dockerfiles for build environments
9. Supported Operating Systems and Ecosystem tests
10. MariaDB Buildbot workflow
11. Challenges encountered and their solutions
12. Conclusion

Introduction

In this article, we will Delve into the details of the new MariaDB Buildbot developed by the MariaDB Foundation. We will begin by providing an overview of the Continuous Integration framework and defining the keywords that will be used throughout the article. We will then discuss the schematic of the Buildbot framework, including the multi-master setup and the use of Docker Latent Workers. Next, we will explore the advantages of using Docker Latent Workers and delve into the specifics of Dockerfiles for different build environments. We will also take a look at the supported Operating Systems and ecosystem tests. Moving forward, we will explain the workflow of the MariaDB Buildbot, highlighting the different steps involved. Finally, we will address the challenges encountered during the development of the Buildbot and the solutions implemented to overcome them.

Overview of the Continuous Integration Framework

The Continuous Integration (CI) framework used in MariaDB's Buildbot is a robust system that ensures that any changes made to the MariaDB Server GitHub repository are thoroughly tested and built. It consists of several components, including the Build Master, Buildbot Worker, and Docker Latent Workers. The Build Master is responsible for detecting changes in the repository and scheduling builds, while the Buildbot Worker handles the actual execution of the builds. By using Docker Latent Workers, the Buildbot achieves a scalable and efficient infrastructure for running builds in isolated environments.

Keywords and Definitions

Before diving into the details, let's define some keywords and concepts that will be used throughout this article.

1. Changes/Repository: Refers to source code changes (commits) that are pushed to the MariaDB Server GitHub repository.
2. Build Master: The main Buildbot process running on a dedicated machine, responsible for detecting changes in the repository and scheduling builds.
3. Build: A defined configuration that consists of a sequence of steps for a particular build, such as getting the source code, compiling the MariaDB Server, and running tests.
4. Buildbot Worker: The process that handles the actual execution of the builds. It runs on a dedicated worker machine and receives instructions from the Build Master.
5. Docker Latent Workers: Instead of having a buildbot-worker continuously running on a dedicated physical machine, Docker Latent Workers utilize Docker containers to start a buildbot-worker process only when needed. Each build starts a separate container.

Schematic of the Buildbot Framework

To better understand the architecture of the MariaDB Buildbot framework, let's take a look at an overall schematic. The Buildbot is configured in a multi-master setup, with two running master processes. One master is dedicated to the user interface, while the other handles the detection of changes in the repository and scheduling of builds. This setup ensures that the user interface remains responsive regardless of the number of running builds. Whenever a push is made to the MariaDB Server repository, it is detected by the Buildbot Master, which then schedules the appropriate builds.

Multi-master Setup for Buildbot

The multi-master setup used in the MariaDB Buildbot framework involves having dedicated master processes for different tasks. One master is responsible for handling the user interface, querying the buildbot database, and presenting the information on the buildbot webpage. The other master takes care of detecting changes in the main repository and scheduling builds accordingly. Both master processes run on the same machine to ensure responsiveness in the user interface. This setup allows for efficient handling of a large number of running builds without compromising the user experience.

Docker Latent Workers

In the MariaDB Buildbot framework, Docker Latent Workers are used to optimize the execution of builds by running buildbot-worker processes inside Docker containers. Instead of having a buildbot-worker process continuously running on a dedicated physical machine, Docker Latent Workers start a Docker container on a remote machine for each build. This container is specifically configured to run the buildbot-worker process and can receive instructions from the Build Master. By using latent workers, there is no continuous buildbot-worker process running on the worker machine, resulting in a more scalable and efficient system.

Advantages of using Docker Latent Workers

Using Docker Latent Workers in the MariaDB Buildbot framework offers several advantages. Firstly, it allows for easy definition and management of different test environments using Dockerfiles. Each Dockerfile specifies the operating system, MariaDB Server build dependencies, and the buildbot-worker Python Package. This flexibility makes it simple to view, deploy, or recreate specific build environments on any machine, facilitating the debugging process. Additionally, Docker Latent Workers enable the straightforward addition of new physical machines to the infrastructure. With only Docker installation required, no further configurations on the worker machines are needed. Furthermore, by starting a new Docker container for each build, a clean environment is always guaranteed, ensuring consistent and reliable test results.

Dockerfiles for Build Environments

Each build environment in the MariaDB Buildbot framework is defined using a Dockerfile. The Dockerfile is divided into four parts. Firstly, the operating system is selected by pulling the correct Docker image from Docker Hub. Then, the MariaDB Server build dependencies are installed, which can be a time-consuming process due to potential repository issues or missing packages. The next step involves installing the buildbot-worker Python package. Finally, the buildbot-worker process is run to establish the connection to the Build Master and prepare for the build. With the use of Dockerfiles, around 50 different environments for different platforms and operating systems have been created and tested.

Supported Operating Systems and Ecosystem Tests

The MariaDB Buildbot framework supports a wide range of operating systems for testing purposes. These systems include various Linux distributions, both community and enterprise versions. Additionally, Windows can also run inside a container, as long as the host system is also running Windows. The Buildbot is also capable of running ecosystem tests, which involve testing frameworks such as PHP against the latest built MariaDB Server version. The framework is continuously expanding, and more ecosystem tests are being added to ensure comprehensive testing across different environments.

MariaDB Buildbot Workflow

The workflow of the MariaDB Buildbot framework involves several steps to ensure proper testing and build processes. The first step is triggered whenever a new change is detected in the MariaDB Server repository. This step involves creating a source tarball and triggering the bintar builds. The source tarball is then used by subsequent builds. In step three, bintars are created by fetching the source tarball, compiling the code, and running tests. The resulting bintar is saved, and package creation builds and ecosystem builds are triggered. The fourth step includes package creation and ecosystem test builds that may run in Parallel. Package creation builds involve fetching the source, creating the packages, saving them, and triggering installation builds and ecosystem tests. The fifth and final step involves testing the installation of previously created packages and performing upgrade tests to ensure smooth upgrades to the latest development version of the MariaDB Server.

Challenges encountered and their solutions

Throughout the development and implementation of the MariaDB Buildbot framework, several challenges were encountered. One of the first issues was related to misconfiguration of the Buildbot, resulting in file mixups during compilation. This issue was resolved by either stopping the Docker container after each build or ensuring proper manual cleanup. Another challenge involved the killing of the mysql-test-run process by SIGPIPE, causing intermittent failures. The root cause of this issue was a Buildbot misconfiguration that was resolved by specifying UTF-8 encoding for both the master and worker processes. Additionally, challenges related to interface responsiveness, build time variation, and file transfer time were identified and addressed through optimizations, multi-master configurations, and changes in data transfer methods.

Conclusion

In conclusion, the new MariaDB Buildbot developed by the MariaDB Foundation provides a robust Continuous Integration framework for testing and building changes made to the MariaDB Server repository. Through the use of Docker Latent Workers, Dockerfiles, and a multi-master setup, the Buildbot offers scalability, efficiency, and ease of use. Despite the challenges encountered during its development, the Buildbot team successfully implemented solutions to ensure smooth operations and reliable test results. The MariaDB Buildbot framework continues to evolve, constantly adding support for new operating systems and ecosystem tests, and aims to provide a seamless experience for developers and contributors in the MariaDB community.

FAQ

Q: What are the benefits of using Docker Latent Workers in the MariaDB Buildbot framework? A: Docker Latent Workers offer a range of benefits, such as easy definition and management of test environments using Dockerfiles, scalability through the use of containers, quick provisioning of clean environments for each build, and the ability to add new physical machines to the infrastructure with minimal configuration requirements.

Q: How many different build environments are currently being tested in the MariaDB Buildbot framework? A: The MariaDB Buildbot framework currently supports around 50 different build environments for various platforms and operating systems. These environments are being continuously tested to ensure compatibility and reliability.

Q: Can the MariaDB Buildbot run tests on Windows operating systems? A: Yes, the MariaDB Buildbot can run tests on Windows operating systems. However, it does require the host system to also run on Windows.

Q: How does the MariaDB Buildbot handle large log files generated during builds? A: To handle large log files, the MariaDB Buildbot limits the amount of logs kept in the Build Master. Only the test results are saved as buildbot logs, while other log files are manually transferred. This helps optimize build time and reduces the strain on the Build Master.

Q: How does the multi-master setup in the MariaDB Buildbot address the issue of interface responsiveness? A: The multi-master setup in the MariaDB Buildbot separates the user interface from the build handling processes, ensuring that the interface remains responsive regardless of the number of running builds. The dedicated master for the user interface handles all interface requests, providing a constant response time.