Search site

Zero-Code OpenTelemetry for Go: Runtime Instrumentation with OBI vs. Compile-Time Instrumentation with Otelc

Updated on: August 25, 2026

At Sematext we’ve been using Go for probably about a decade. But we didn’t start instrumenting it with OpenTelemetry until earlier this year.

Go has historically had a relatively straightforward but hands-on OpenTelemetry instrumentation model: add the OpenTelemetry SDK, initialize it, instrument the libraries you use, and create custom spans where application-specific context matters.

That approach still gives you the most control. But it is no longer the only practical option.

The manual instrumentation approach requires so much work from engineers that we decided to create a general AI Skill for instrumenting applications with OpenTelemetry SDK, regardless of the runtime/SDK.

Two newer approaches can instrument Go applications with little or no manual source-code instrumentation:

OpenTelemetry eBPF Instrumentation (OBI) instruments applications at runtime.
OpenTelemetry Go Compile-Time Instrumentation (otelc) injects instrumentation during the build.

Both are part of the OpenTelemetry ecosystem. Both can produce OpenTelemetry telemetry without requiring developers to manually wrap every HTTP handler, database call, gRPC client, or messaging operation.

Both can be described as “zero-code instrumentation”, but they solve the problem at completely different points in the software lifecycle.

OBI asks:

How can we observe this application without changing or rebuilding it?

Otelc asks:

How can we build this application with instrumentation already inside it without manually modifying its source code?

This article looks at both approaches from the perspective of a developer, SRE, DevOps engineer, or engineering manager who needs to make a practical decision about instrumenting a Go application.

The traditional way to instrument Go with OpenTelemetry

The conventional way to instrument a Go application is to explicitly add OpenTelemetry support to the application using the SDK for Go.

At a high level, that usually means:

Adding the OpenTelemetry Go API and SDK.
Configuring a TracerProvider.
Configuring exporters.
Instrumenting libraries such as HTTP servers and clients, gRPC, databases, and messaging clients.
Adding custom spans around important application operations.

A simplified example might look like this:

tracer := otel.Tracer("checkout")
ctx, span := tracer.Start(ctx, "reserve_inventory")
defer span.End()
if err := inventory.Reserve(ctx, order); err != nil {
  span.RecordError(err)
  return err
}

For a more complete example, see our Gin service instrumentation example: https://github.com/sematext/sematext-otel-onboarding/tree/main/go

This model has important advantages:

You explicitly control where spans start and end.
You can attach application-specific attributes.
You can model important business operations.
You can decide what should and should not become telemetry.

The downside is obvious: instrumentation becomes part of the application.

For a sufficiently large system, that can mean touching many services, maintaining instrumentation dependencies, reviewing instrumentation changes, and deciding how deeply each library and operation should be instrumented. This is the type of stuff we faced at Sematext when we said “OK, let’s go and instrument all our Go services now”.

A useful way to think about the landscape is:

OBI and Otelc occupy different places on that spectrum.

Note that manual instrumentation is far from obsolete, you and your team can still choose this approach. It’s just that the zero-code approaches provide another layer of options. As a matter of fact, as you read this article you will learn that manual instrumentation is still critical in certain situations and complements the auto-instrumentation approaches.

Why zero-code instrumentation is harder in Go

Automatic instrumentation is relatively familiar in languages with highly dynamic runtimes.

Java agents can modify or intercept bytecode. Python can wrap functions dynamically. Other runtimes provide mechanisms that make it possible to insert instrumentation after an application has been built.

But Go is different. A Go application is typically compiled into a native binary. Once the binary exists, there is no general-purpose equivalent of loading a Java agent and rewriting the application’s bytecode.

That creates a fundamental choice for automatic instrumentation:

Do you instrument the process from outside, or do you modify the application during compilation?

OBI takes the first approach and Otelc takes the second.

The architectures look like this.

OBI: runtime instrumentation

Otelc: compile-time instrumentation

This difference affects deployment, security, portability, ownership, and operational complexity.

Let’s look at each approach in detail.

OBI: Instrumenting a running Go process

OpenTelemetry eBPF Instrumentation (OBI) is the OpenTelemetry project’s eBPF-based automatic instrumentation technology.

It was originally based on technology from Grafana Beyla and is now developed as an OpenTelemetry project. OBI’s first OpenTelemetry release was announced in late 2025, and the project has continued to evolve since then.

We are big fans of eBPF at Sematext and have been using it for 10+ years now. When we first started instrumenting our Go applications we chose the OBI approach and included it in our OTel examples for instrumenting Go applications.

The central idea is simple:

Run instrumentation outside the application process and observe the application while it runs.

Unlike a traditional OpenTelemetry SDK integration, OBI does not require adding OpenTelemetry code to the application binary.

How OBI works

OBI uses eBPF to observe application and system behavior. Depending on what is being instrumented, it can capture activity at protocol boundaries and use language-specific techniques, including user-space probes, or uprobes, for supported Go instrumentation.

A simplified flow looks like this:

         Client
           │
           │ HTTP request
           ▼
┌──────────────────────┐
│      Go service      │
│                      │
│   net/http handler   │◄───── OBI observes
│          │           │       supported operations
│          ▼           │
│     application      │
│       logic          │
│          │           │
│          ▼           │
│    database/sql      │◄───── OBI observes
│                      │
└──────────────────────┘
           │
           │ SQL
           ▼
       PostgreSQL
           │
           ▼
   ┌───────────────┐
   │      OBI      │
   └───────┬───────┘
           │
           ▼
    Trace + metrics

OBI can be deployed as:

a standalone process,
a Docker container,
a Kubernetes sidecar,
or a Kubernetes DaemonSet.

The exact deployment model depends on how broadly you want to instrument the environment.

What can OBI instrument in Go?

OBI supports a combination of protocol-level and Go library-level instrumentation.

The current OpenTelemetry documentation lists Go support for technologies including:

net/http
HTTP/2
gorilla/mux
Gin
gRPC
database/sql
MySQL drivers
PostgreSQL drivers
Redis
Kafka
Sarama
pgx

Compatibility depends on the specific library and version. For example, Go library-level instrumentation is documented for Go 1.17+, while some context propagation capabilities require Go 1.18+.

This is an important practical point. OBI is not simply “watching packets.”

For supported Go libraries, it can use Go-specific instrumentation to capture richer application behavior.

At the same time, it remains fundamentally an out-of-process instrumentation system.

What using OBI looks like

Imagine you have an existing Go service, say payment-api, and that it is already running in production.

As such, you may not want to:

modify its source code,
change its dependencies,
rebuild it,
or restart it purely to add instrumentation.

OBI can be deployed separately and configured to discover or target the application. This is particularly useful for existing workloads.

OBI’s runtime model is one of its strongest characteristics: it can observe applications without making OpenTelemetry instrumentation part of the application’s build artifact. The OpenTelemetry project describes OBI as out-of-process instrumentation that can provide telemetry without application code changes or application restarts in supported scenarios.

Advantages of OBI

1. No source-code changes

The obvious benefit is that developers do not need to edit the application.

There is no need to:

git clone --> add instrumentation --> test instrumentation --> commit --> build --> deploy

Instead, instrumentation can be introduced independently of the application code.

This is particularly attractive when:

the application is maintained by another team,
source code is unavailable,
the application is legacy,
you want to instrument an existing fleet,
or the platform team owns observability deployment.

2. No application rebuild

OBI can instrument supported workloads without requiring you to produce a new binary. This is one of the clearest differences between OBI and Otelc.

Otelc, on the other hand, requires the ability to run the application through an instrumented build process.

3. Instrumentation can be centrally operated

OBI is well suited to an infrastructure-oriented operating model. For example, the platform team could choose to deploy OBI as a Kubernetes DaemonSet and have it handle the instrumentation of Go, Java, and Python applications. In other words, the application teams do not necessarily have to independently add and maintain instrumentation.

That can be valuable in organizations with many teams and inconsistent OpenTelemetry adoption.

4. It is not Go-specific

OBI can observe applications written in multiple languages. That means an SRE or platform engineering team can use a common instrumentation mechanism across a heterogeneous environment. For organizations operating Go, Java, Python, Node.js, NGINX, and other workloads, that can simplify initial telemetry coverage. This is one of the OBI aspects that we have benefited from at Sematext, as some of our legacy code still uses Java/Kotlin and Node.js.

The limitations and trade-offs of OBI

1. OBI depends on the runtime environment

OBI is fundamentally a Linux and eBPF-based technology.

That means your deployment environment must support the capabilities OBI needs.

The OpenTelemetry documentation describes OBI as a Linux process that can inspect other running processes and requires elevated privileges or the appropriate Linux capabilities, depending on deployment and configuration.

In containers and Kubernetes, this can become an architectural decision rather than a simple configuration change.

For example, some OBI deployments require or may use:

privileged containers,
CAP_SYS_ADMIN,
CAP_PERFMON,
host or shared process namespaces,
access to /proc.

The exact requirements depend on the instrumentation and deployment model. Recent kernel security changes can also affect Go instrumentation because OBI uses uprobes for Go-specific instrumentation.

This does not mean OBI is inherently unsuitable for production.It means that the security model must be evaluated by the platform team.

Of course, if your applications don’t run on Linux then OBI is not an option for you at all until other platforms, like Windows, get the needed eBPF support. See https://github.com/microsoft/ebpf-for-windows for what Microsoft is doing about that for Windows.

2. Coverage depends on what OBI understands

OBI can automatically observe supported protocols and libraries, but it does not automatically understand arbitrary application code.

Consider:

func CalculateEnterpriseDiscount(
    customer Customer,
    contract Contract,
) (Discount, error) {
    // 500 lines of business logic
}

There is no general way for an external runtime observer to know that this function represents an important business operation. OBI can show what happens around it:

HTTP request
    │
    ├── database query
    ├── Redis lookup
    ├── gRPC call
    └── Kafka publish

But it cannot automatically know that CalculateEnterpriseDiscount is an important domain-level operation.

3. Automatic service names and routes may need tuning

Because OBI observes applications externally, automatically derived service names, routes, and URLs may not always match how your organization wants to identify services.

The OpenTelemetry documentation specifically calls out route configuration and decoration as something that should be reviewed when generating traces with OBI.

That means you should validate the resulting telemetry rather than assuming that automatic discovery will always produce exactly the naming and cardinality you want.

Otelc: Instrumenting Go during compilation

The second approach moves instrumentation from runtime to build time.

Otelc, the OpenTelemetry Go Compile-Time Instrumentation tool, modifies the Go build process so that supported instrumentation is injected while the application is being compiled.

The resulting application binary contains the instrumentation.

The normal Go build looks like this:

Source code ──► go build ──► Go binary

While with Otelc it looks like this:

Source code ──► otelc + go build ──► Instrumented Go binary

So if you choose Otelc you will not need to change the application source code, but you will need to alter the build process.

How Otelc works

According to the OpenTelemetry Go compile-time instrumentation documentation, Otelc:

Intercepts compilation using the Go toolchain’s -toolexec mechanism.
Matches packages and functions against instrumentation rules.
Injects lightweight hook points.
Links those hooks to OpenTelemetry instrumentation code.

The resulting binary contains the instrumentation, so there is no separate runtime instrumentation agent that needs to attach to the process. Operationally, this is simpler because there is no additional moving piece running in production.

Visually things look like this:

Go application (source) ──► Otelc (match rules, inject hooks, link OTel code) ──► Instrumented Go binary ──► OTLP

The Otelc project uses techniques including trampoline code injection and function hook mechanisms to connect instrumented functions with OpenTelemetry logic.

What does using Otelc look like?

A simple workflow can look like:

otelc go build -o myapp .

Alternatively, Otelc can be integrated with the standard Go toolchain using -toolexec.

A documented pattern is:

otelc setup
export GOFLAGS="${GOFLAGS} '-toolexec=otelc toolexec'"
go build -o myapp .

This can be useful when a build command is controlled by an existing Makefile, CI system, or another build tool.

Otelc can also be installed as a Go tool dependency in supported Go versions, allowing builds such as:

go tool otelc go build -o myapp .

Otelc also supports generating or maintaining instrumentation configuration based on the application’s dependency graph.

What can Otelc instrument?

The currently documented set of supported instrumentation includes:

net/http
gRPC
database/sql
Gin
Redis
MongoDB
Kubernetes client-go
OpenAI Go SDK
Anthropic Go SDK
Kafka
AWS SDK for Go v2
selected logging libraries for trace/span correlation

The supported set will continue to evolve, so it is worth checking the project’s current supported-library documentation before choosing it for a specific application.

Advantages of Otelc

1. No manual instrumentation changes

Developers do not have to manually add instrumentation to every supported library boundary.

The source can remain:

http.HandleFunc("/checkout", checkoutHandler)

rather than becoming:

handler := otelhttp.NewHandler(
    http.HandlerFunc(checkoutHandler),
    "checkout",
)
http.Handle("/checkout", handler)

The instrumentation is introduced during compilation instead.

2. No privileged runtime instrumentation process

Once the application has been built, there is no eBPF process that needs to attach to it.

This can make Otelc attractive in environments where:

privileged containers are prohibited,
eBPF is unavailable,
security policy restricts process instrumentation,
or platform teams do not want observability software attaching to production workloads.

The OpenTelemetry documentation explicitly identifies this as a use case for compile-time instrumentation.

3. Instrumentation can reach supported dependencies

Because Otelc participates in the compilation process, it can instrument supported third-party dependencies that are part of the application’s build. That is useful when you use a library that you do not own but still want to instrument.

For example:

Your application
       │
       ├── Gin
       ├── gRPC
       ├── database/sql
       ├── Redis
       └── AWS SDK

Otelc can apply instrumentation rules to supported parts of that dependency graph without requiring you to fork or edit those dependencies.

4. The build artifact contains the instrumentation

This changes who owns the operational problem.

With OBI: application deployment + runtime instrumentation deployment

With Otelc: only the instrumented application artifact

The instrumentation becomes part of the software artifact produced by the build. That can fit naturally into organizations where application teams already own their build and deployment pipelines.

The limitations and trade-offs of Otelc

Nothing in this world seems to come without downsides… let’s look at Otelc’s cons.

1. You must control the build

Otelc requires access to the build process because that is where instrumentation is introduced. This is the most important limitation. If you have a precompiled production binary but cannot rebuild it, Otelc is simply not an option.

2. The build pipeline becomes part of the instrumentation architecture

Adding Otelc is not the same as adding another environment variable.

You now need to think about:

local developer builds,
CI builds,
release builds,
test builds,
reproducibility,
dependency management,
cross-compilation,
monorepos,
and build caching.

The good news is that Otelc is designed to work with the normal Go build workflow and documents approaches for integrating through go tool, direct build wrapping, and -toolexec.

But this should still be treated as a build-system change and tested accordingly.

3. Coverage is limited to available instrumentation

Like OBI, Otelc does not automatically understand every Go package. It needs instrumentation rules for the libraries and functions you want to observe.

If you use some framework that is not yet supported by the instrumentation, Otelc will not magically infer its semantics.

You may need to:

add manual instrumentation,
create instrumentation for that library,
or accept that the library is not automatically traced.

Luckily, the Otelc project includes an instrumentation model and documentation for adding support for additional libraries, so this scenario can be handled.

4. Automatic instrumentation is still not the same as manual application instrumentation

Otelc can add spans around supported framework and library operations. It does not automatically know which parts of your business logic are important.

For example:

POST /checkout
       │
       ▼
ValidateOrder
       │
       ├── ReserveInventory
       │
       ├── CalculateDiscount
       │
       ├── ProcessPayment
       │
       └── CreateShipment

Automatic instrumentation may produce excellent visibility into:

HTTP server span
       │
       ├── SQL query
       ├── Redis operation
       ├── HTTP call to payment provider
       └── Kafka publish

But it may not tell you how much time was spent specifically in CalculateDiscount unless you explicitly instrument that operation.

OBI vs. Otelc: side-by-side comparison

The following table summarizes the practical differences.

Characteristic	OBI	Otelc
Instrumentation point	Runtime	Build time
Primary mechanism	eBPF, protocol observation, and language-specific probes such as uprobes	Go compiler/toolchain integration and injected instrumentation hooks
Source-code changes	None required	None required
Application rebuild required	No	Yes
Can instrument an existing binary	Yes, in supported environments	No
Can observe an already-running process	Yes	No
Requires build pipeline changes	No, not necessarily	Yes
Requires runtime instrumentation software	Yes	No separate attach-time agent
Requires Linux/eBPF support	Yes	No eBPF dependency
Requires elevated runtime privileges	Often requires privileged operation or specific Linux capabilities, depending on deployment	No eBPF-related runtime privileges
Can instrument supported Go libraries	Yes	Yes
Can instrument supported third-party dependencies	Yes, depending on supported protocol/library instrumentation	Yes, when supported instrumentation rules exist
Works when source code is unavailable	Potentially, yes	Only if you can rebuild from source
Works with precompiled binaries	Yes	No
Cross-language use	Yes	Primarily Go
Typical operational owner	Platform engineering, SRE, DevOps	Application engineering and/or CI/CD/platform engineering
Best fit	Existing workloads and centralized runtime instrumentation	Applications where you control the Go build
Business-level custom spans	Requires additional/manual instrumentation	Requires additional/manual instrumentation
Instrumentation deployment	Separate from the application artifact	Baked into the build artifact

OBI vs. Otelc: how to choose which one to use

A simple decision matrix

A quick way to determine if you should be considering OBI or Otelc is by considering the following scenarios and asking a few questions.

If your starting point is:

“I have a binary already running and I don’t want to rebuild it.”

Start with OBI.

Existing binary?
      │
      ├── Yes ──► OBI is the practical zero-code option
      │
      └── No

“I control the build but don’t want to modify the application source.”

Look at Otelc.

Control the Go build?
      │
      ├── Yes ──► Otelc is a strong candidate
      │
      └── No ──► Consider OBI

“I cannot run privileged instrumentation in production.”

Look at Otelc.

eBPF / runtime privileges allowed?
      │
      ├── No ──► Otelc
      │
      └── Yes ──► OBI or Otelc

“I need to instrument services in several languages.”

OBI may provide a better common platform-level approach.

Go + Java + Python + Node.js
             │
             ▼
            OBI

“I need to instrument important internal business operations.”

Neither automatic approach completely solves the problem. You will probably still want manual instrumentation. Ooops! 😉

A practical decision tree

Here is another approach, a decision tree, that will help you quickly see which approach is more suitable.

If you are deciding how to instrument a Go application, start with this:

                      ┌──────────────────────┐
                      │ Need OpenTelemetry?  │
                      └──────────┬───────────┘
                                 │
                                 ▼
                  ┌────────────────────────────────┐
                  │ Can you modify the application │
                  │ source code?                   │
                  └──────────┬─────────────────────┘
                             │
                ┌────────────┴────────────┐
                │                         │
               Yes                        No
                │                         │
                ▼                         ▼
        Manual instrumentation     Can you rebuild
        is available                the application?
                                      │
                           ┌──────────┴──────────┐
                           │                     │
                          Yes                    No
                           │                     │
                           ▼                     ▼
                     Consider Otelc        Consider OBI
                           │                     │
                           ▼                     ▼
                    Do you need             Does your runtime
                    domain-level spans?     support OBI/eBPF?
                           │                     │
                           ▼                     ▼
                  Add manual spans        Deploy and validate
                  where they matter       supported coverage

In practice, the decision often reduces to four questions.

Question 1: Do I control the build?

If yes, Otelc becomes an option.

If no, it does not.

Question 2: Can I run eBPF instrumentation in production?

If yes, OBI becomes an option.

If no, Otelc may be easier operationally.

Question 3: Do I need to instrument existing binaries?

If yes, OBI is the more natural fit.

Question 4: How much application-specific context do I need?

If the answer is “a lot,” neither zero-code approach is likely to be sufficient by itself. Plan for some manual instrumentation.

Why automatic instrumentation does not eliminate the need for manual instrumentation

This is perhaps the most important point in this entire article. It is tempting to think of automatic instrumentation as a replacement for manual instrumentation. In our experience, it usually is not.

Automatic instrumentation and manual instrumentation solve different problems.

Automatic instrumentation is excellent at finding and instrumenting common technical boundaries:

HTTP requests,
gRPC calls,
database queries,
Redis operations,
messaging operations,
cloud SDK calls,
and other supported libraries.

Manual instrumentation is where you describe what your application actually does.

Consider a checkout service.

Automatic instrumentation may give you:

POST /checkout                           820 ms
│
├── SELECT customer                      12 ms
├── SELECT inventory                     18 ms
├── Redis GET                             3 ms
├── POST payment-provider               410 ms
└── Kafka publish                         8 ms

This is indeed already extremely useful and you should aim for this as your first step.

But your engineering team may care about something different:

Checkout
│
├── ValidateOrder
├── ReserveInventory
├── CalculateDiscount
├── ProcessPayment
└── CreateShipment

Those are domain operations. Neither OBI or Otelc can reliably infer that these operations are important simply by observing technical behavior.

The most effective approach is often a hybrid.

                HTTP request
                       │
                       ▼
        ┌──────────────────────────┐
        │ Automatic instrumentation│
        └────────────┬─────────────┘
                     │
              Application code
                     │
                     ▼
        ┌─────────────────────────┐
        │   Manual business spans │
        │                         │
        │ ReserveInventory        │
        │ CalculateDiscount       │
        │ ProcessPayment          │
        └────────────┬────────────┘
                     │
                     ▼
        ┌──────────────────────────┐
        │ Automatic instrumentation│
        │ SQL / Redis / gRPC / etc │
        └──────────────────────────┘

For OBI specifically, OpenTelemetry also provides the Go Instrumentation Auto SDK, which is intended to help integrate manually created spans with eBPF-generated spans and shared trace context.

That makes the hybrid model especially relevant:

Use automatic instrumentation for broad baseline coverage. Add manual spans only where application-specific context materially improves observability.

This avoids two extremes:

Extreme 1: Instrument nothing automatically

Every team has to manually instrument every HTTP framework, database driver, messaging library, and client.

Extreme 2: Instrument everything automatically and assume the result is sufficient

You get infrastructure-level telemetry but may lack the domain context required to answer questions such as:

Why is checkout slow?
Which business operation failed?
How much time is spent calculating pricing?
Which customer workflow is affected?
Did the payment provider fail, or did our own validation logic reject the request?

The best observability, as I hope I’ve illustrated so far in this article, usually combines the two approaches.

Conclusion

OBI and Otelc represent two fundamentally different approaches to zero-code OpenTelemetry instrumentation for Go.

OBI instruments from the outside.

It is attractive when you want to observe applications that already exist, especially when rebuilding or modifying them is difficult. It can be deployed independently of the application and can support a centralized, platform-owned instrumentation model.

Otelc instruments from the inside—during the build.

It is attractive when you control the Go build process and want supported OpenTelemetry instrumentation to become part of the resulting binary without manually modifying application source code. It also avoids the need for a privileged runtime eBPF instrumentation process.

Neither approach eliminates the value of manual instrumentation.

A practical architecture for many teams will look like this:

┌─────────────────────────────────────────────┐
│           Automatic instrumentation         │
│                                             │
│   OBI or Otelc                              │
│                                             │
│   HTTP • gRPC • SQL • Redis • Kafka • etc.  │
└───────────────────────┬─────────────────────┘
                        │
                        ▼
┌─────────────────────────────────────────────┐
│           Manual instrumentation            │
│                                             │
│   Business operations                       │
│   Domain-specific spans                     │
│   Important attributes                      │
│   High-value application context            │
└───────────────────────┬─────────────────────┘
                        │
                        ▼
              OpenTelemetry backend

The practical goal should not necessarily be to choose one instrumentation method and use it everywhere. Instead, choose the method that best fits the part of the system you are trying to observe.

Need visibility into existing workloads without rebuilding them? Start with OBI.
Control the Go build and want instrumentation baked into the binary? Look at Otelc.
Need detailed visibility into business operations? Add manual instrumentation.
Need all three? A hybrid approach may be the most useful architecture.

For implementation details and current compatibility information, start with the authoritative project documentation:

Start Free Trial