Testing Services at Scale: The Secrets of Automation

Testing Services at Scale: The Secrets of Automation

Table of Contents:

1. Introduction
2. The Importance of Testing Services
3. Engineering Culture at Facebook
4. The Infrastructure of Facebook
5. Unit and Integration Testing
6. Challenges in Service and Integration Testing
7. The Integration Testing Framework at Facebook
8. Automating Testing with Fuzzing
9. The Synergy between Fuzzing and Integration Tests
10. Automating Integration First Testing
11. Improving First Testing at Facebook
12. Conclusion

Automating Testing with Fuzzing

Automated testing plays a crucial role in ensuring the reliability and functionality of software systems. At Facebook, various tools are used for automated testing, such as Infer for static analysis and fuzzing for detecting abnormal behavior. Fuzzing involves repeatedly testing a program with different inputs to uncover bugs or crashes. However, the process of fuzzing typically requires manual steps, including carving out a test target and writing a fuzz harness.

Facebook realized that the most synergy between fuzzing and integration tests could be achieved by testing the entire service instead of a specific test target. This shift allowed for the elimination of manual steps and the automation of the testing process. By leveraging the testing framework used by all services at Facebook, which supports reflection, random inputs could be Shaped into correct types and sent to the service for testing. Moreover, the data sent to the service over the network is considered fair game, meaning that the service must respect its contract regardless of the input data.

This integration-first testing approach at Facebook combines the benefits of automated fuzzing with the comprehensiveness of integration testing. It allows for the discovery of corner cases and deeper code paths, resulting in higher test coverage and improved software quality. Additionally, by recording and sanitizing requests sent to production instances of a service, realistic and constrained data can be used as seeds for fuzzing, further enhancing the effectiveness of the testing process.

Facebook has successfully used this integrated approach for over a year, automatically fuzzing services in the background and filing bug reports when issues are discovered. The benefits of this methodology have been evident in the improved debugging experience for service owners and the increased coverage achieved through the automated process. However, Facebook continues to focus on improving their fuzzing and integration testing techniques, aiming to enhance isolation, leverage coverage to identify bottlenecks, and ultimately provide service owners with confidence in deploying their services to production with only unit tests and integration fuzz tests.

In conclusion, the automation of testing processes, particularly through the integration of fuzzing and integration testing, has proven to be a valuable approach at Facebook. By eliminating manual steps and leveraging the testing framework, comprehensive and efficient testing can be achieved, resulting in higher software quality and increased confidence in production deployments.