"As our production systems, and especially our open web traffic system, have grown over the years, the complexity of testing them has also increased. Although automated tests have some benefits, we could not test all use cases of the traffic, and had no way to predict edge case bugs in a testing environment. After encountering several major production incidents, the need for a solution for both functional and non-functional testing environments was obvious."
"As our traffic system at DoubleVerify is designed to process billions of requests per day, creating such non-production environments is a significant challenge. An integral part of this type of environment is replaying real traffic from the production environment to a testing and integration environment. In the above example, we can see that we sent a configurable percentage of the incoming event stream to a Kafka topic and replicated it via the "mirroring" tool. We used Mirror Maker, but you can use any mirroring tool you choose and even implement a consumer-producer-based tool of your own."
Non-production environments are necessary for testing and integration and present significant challenges when production processes billions of requests per day. Replaying real production traffic into non-production systems enables uncovering edge-case bugs and validating both functional and non-functional behavior. A common pattern is to send a configurable percentage of incoming events to a Kafka topic and replicate it using a mirroring tool such as Mirror Maker, or a custom consumer-producer implementation. Supporting functional tests requires maintaining a stable replay rate matched to non-prod capacity. Supporting non-functional tests requires handling peak rates and system performance characteristics during replay.
Read at Medium
Unable to calculate read time
Collection
[
|
...
]