Connecting OpenTelemetry spans to ongoing chaos experiments creates clear links between the code, your product and engineering teams, and an active chaos experiment. The full context from OpenTelemetry removes the mystery when figuring out "what happened?" during chaos engineering experiments.
| What are you looking to do? |
|---|
| Use existing telemetry from your service |
| See example integrations |
The goal here is to annotate testing data with rich telemetry data so that the developer can quickly isolate issues and improve the system.
- Determine existing event(s) available from your product that can fire a webhook.
- Determine existing metric or tracing formats that are already supported by your product (i.e. OpenTracing, Prometheus exporters).
- Review the OpenTelemetry collector and -contrib respositories: does an integration already exists, or is it possible to use an existing integration?
- If no prior work exists, use the OpenTelemetry Collector Builder to start a new golang project that receives events from your service and annotates relevant OpenTelemetry metrics, logs, and traces.
- Send logs, metrics or traces telemetry to an OpenTelemetry collector that's configured to recieve webhooks from your service using code you wrote or an existing integration.
- Using your new custom collector processor, attach the chaos experiment id to the span when the webhook event trigger is fired, and remove it when the webhook resolution trigger is fired.
- Send webhooks from your product to an OpenTelemetry collector configured for console output. Verify outpiut in the console.
- Optional: verify in an OpenTelemetry production tool of your choice
Make your code usable to as many people as possible! If you're looking for help here, contact us at partnerships@lightstep.com. We'd love to help support you!
| Integration | Description |
|---|---|
| webhookprocessor | OpenTelemetry collector that can receive Gremlin webhooks. |