BrutRB + Otel - sorta works!

General
, ,
1

One of my goals for Brut was to make observabililty/instrumentation built-in via OTel. In fact, I’m trying to make it so that there is no logging, just OTel. I’m not quite there, but OTel is working much better now.

As mentioned previously, OTel is not well documented and most of what’s there is geared toward using OTel as a client and not incorporating it into a framework.

I had it working pretty well for web requests, but really wanted to see a trace where a web requested queued a Sidekiq job, and then the trace included that job’s behavior.

A lot of OTel config says to do use_all, but this is generally a mistake IMO. Configuration code isn’t often edited, so it doesn’t benefit from slick batteries-included DSLs that do it all. It’s often much better to have it explicitly laid out so you can see what’s what and now what you can change and where.

Here is how Sidekiq is set up:

if defined?(OpenTelemetry::Instrumentation::Sidekiq)
  c.use 'OpenTelemetry::Instrumentation::Sidekiq', {
    span_naming: :job_class,
    propagation_style: :child, # <--- default is "link"
  }
else
  SemanticLogger[self.class].info "OpenTelemetry::Instrumentation::Sidekiq is not loaded, so Sidekiq traces will not be captured"
end

The default “propagation style” is “link”, which I think is intende to link the trace that queued a job to the trace that processes it. In practice, this doesn’t seem to actually create the link or the link doesn’t show up (at least Honeycomb and otel-desktop-viewer - OTel is extremely hard to debug). Setting the style to “child” puts the processing trace as a child of the original trace.

[As with many things, I spent weeks putting it off and got it working in like 15 minutes.]

Here’s screenshots from local otel-desktop-viewer showing a web request that leads to a job queue that is then processed:

Pasted image
Pasted image2171×1514 310 KB

Pasted image (2)
Pasted image (2)2147×1393 247 KB

Pasted image (3)
Pasted image (3)2147×914 145 KB

2 Likes
2

That trace of the web request looks great, glad you got it working!

This level of observability would be included OOTB with Brut or just add the components so everyone can wire whatever they need.

3

Yeah, so everything shown up to the queuing of the sidekiq job is included by default (and the Brut dev environment includes otel-desktop-viewer so you can see this in your dev environment).

When you use brut new segment sidekiq to add Sidekiq (which is basically a bit of codegen), that will, as of the next release, properly set up Sidekiq OTel as well, so that would then include the job queuing and processing shown in the diagram.

You can add your own instrumentation as well: Instrumentation and Observability | Brut RB (this is a wrapper/proxy around the OTel ruby lib)

1 Like