Add finish method to synchronous instruments

[atoulme]

Fixes #2232

Changes

Add the ability to finish reporting a synchronous instrument attribute set. The attribute set under that instrument will no longer report.

POCs

open-telemetry/opentelemetry-go#8050
open-telemetry/opentelemetry-java#7792

[dashpole]

One thing that has been asked for is to be able to delete multiple series at once, since callers often don't have access to the complete list of attribute sets they've previously incremented. E.g. remove(http.target=foo) would remove all streams for that http.target. One way to solve this is by matching all attribute sets which don't have the keys provided. Remove without any arguments would match and remove all streams from the instrument. Unfortunately, it wouldn't be backwards-compatible to change this later, so we need to decide if this is important from the start.

The other big question (which we need to resolve in the SDK spec PR), is how this impacts start time handling for cumulative metrics in the SDK. If an attribute set is deleted and recreated, the resulting metric data must have non-overlapping start-end time ranges since the cumulative value has been (presumably) reset. One way to solve this would be to require per-attribute-set start times in the SDK (#4184).

[jack-berg]

A more flexible version of @dashpole's suggestion might look like: remove(Predicate<Attribute> predicate), where the predicate is invoked for each series, with usage in java like:

instrument.remove(attributes -> true); // Remove all series
instrument.remove(attributes -> attribute.get("http.route").equals("/v1/foo/bar")); // Remove all series where http.route=/v1/foo/bar
instrument.remove(attributes -> attributes.get("http.route").startsWith("/v1/foo")); // Remove all series matching pattern http.route=/v1/foo.*

If an attribute set is deleted and recreated, the resulting metric data must have non-overlapping start-end time ranges since the cumulative value has been (presumably) reset. One way to solve this would be to require per-attribute-set start times in the SDK (#4184).

Yeah for example, in java, we always use the same SDK start time for all cumulative series. Whether they see their first data at start time or days later, the start is always the same. I think your suggestion to track per-attribute-set start times in the SDK seems reasonable, but that seems to imply a behavior change from the single constant start time that we currently do java. Are there implications for this? Are there cases where a user with a cumulative backend would be upset to see new series with a start time corresponding to the window where data was first recorded?

[jack-berg]

Also, if I call something like instrument.remove(attributes -> true) to delete all series, its not clear whether I intend to record additional data in the future. If I don't intend to record additional data, then I would probably view the fact that the SDK continues to have memory allocated to the instrument as a memory leak. But the SDK can't free up all the resources for that instrument without knowing for sure that I won't record again.

Makes me wonder if we need a top level instrument level close / remove method, as well as the fine grained method for removing specific series.

[dashpole]

Are there implications for this? Are there cases where a user with a cumulative backend would be upset to see new series with a start time corresponding to the window where data was first recorded?

I don't think it will impact Prometheus in any negative way (@ArthurSens might know more, since he opened the original issue).

Depending on how strict you want to be, it may make it harder to aggregate timeseries with different start timestamps. If you user the earliest start timestamp, you may be missing data, and not produce an accurate cumulative for the entire time range.

[carlosalberto]

cc @jmacd

[atoulme]

Makes me wonder if we need a top level instrument level close / remove method, as well as the fine grained method for removing specific series.

This can be added later and separately from this effort, from what I can tell.

[atoulme]

If an attribute set is deleted and recreated, the resulting metric data must have non-overlapping start-end time ranges since the cumulative value has been (presumably) reset. One way to solve this would be to require per-attribute-set start times in the SDK (#4184).

My concrete use case is tied to a queue system where we report the count of events seen. See https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/ibm-mq-metrics/model/metrics.yaml#L216

When we lose contact with the queue manager, we can no longer report this information. If we get back in contact with the queue manager, we must recreate the time series with the new information, resetting the counter and its start time. We typically alert on delta changes, so we want the time series to be separate.

[dashpole]

we must recreate the time series with the new information, resetting the counter and its start time. We typically alert on delta changes, so we want the time series to be separate.

How do you reset the start time? All SDKs i'm aware of in OTel today set the cumulative start time at instrument creation time, and never reset it.

[ArthurSens]

Are there implications for this? Are there cases where a user with a cumulative backend would be upset to see new series with a start time corresponding to the window where data was first recorded?

Actually, we would be happier to see this :) I've opened #4184 a long time ago, but never found the time to continue working on it. A start time per time series would help us provide more accurate increase rates.

[atoulme]

Since this is now sponsored, I am marking this PR ready for review. The discussion continues.

[atoulme]

we must recreate the time series with the new information, resetting the counter and its start time. We typically alert on delta changes, so we want the time series to be separate.

How do you reset the start time? All SDKs i'm aware of in OTel today set the cumulative start time at instrument creation time, and never reset it.

Let's put this in as a requirement and try it out in the POCs, see how we fare.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[jmacd]

@atoulme

My concrete use case is tied to a queue system where we report the count of events seen. See https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/ibm-mq-metrics/model/metrics.yaml#L216

When we lose contact with the queue manager, we can no longer report this information. If we get back in contact with the queue manager, we must recreate the time series with the new information, resetting the counter and its start time. We typically alert on delta changes, so we want the time series to be separate.

This issue has been raised and resisted a number of times in OpenTelemetry. I understand there is still an unanswered need, but I do not like the verb remove as the API method name, it's not clearly "removing" anything; it's not de-registering the instrument. We are not trying to erase the memory of the instrument, we're trying to get series out of memory. We need to report final measurements, "seal" the timeseries in some manner, and then forget about the data. If I could choose the verb for this action, it's "finish", it means "flush and forget". I think the idea of passing a predicate to select series for finishing makes sense.

Consumers should receive the correct finalized value of these series. As @ArthurSens points out in #4184, what we need is a specification for how the data should be transmitted so that the ending of a series is clear. In Prometheus, we have the NaN value, and in OTel we have the missing data point flag, but we've never specified how to set that flag. I would like to see a specification that dictates SDKs have to remember the "finishing" series long enough to send the NaN/missing-data-flag to each reader at least once, otherwise that reader would lose information. Then, to answer #4184, we need to specify that new series must be created with a start time >= the Nan/missing-data-flag previously issued for the same series.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[github-actions]

Closed as inactive. Feel free to reopen if this PR is still being worked on.

[atoulme]

Can I please ask for a maintainer to reopen the pull request? Thanks.

[atoulme]

@atoulme

My concrete use case is tied to a queue system where we report the count of events seen. See https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/ibm-mq-metrics/model/metrics.yaml#L216

When we lose contact with the queue manager, we can no longer report this information. If we get back in contact with the queue manager, we must recreate the time series with the new information, resetting the counter and its start time. We typically alert on delta changes, so we want the time series to be separate.

This issue has been raised and resisted a number of times in OpenTelemetry. I understand there is still an unanswered need, but I do not like the verb remove as the API method name, it's not clearly "removing" anything; it's not de-registering the instrument. We are not trying to erase the memory of the instrument, we're trying to get series out of memory. We need to report final measurements, "seal" the timeseries in some manner, and then forget about the data. If I could choose the verb for this action, it's "finish", it means "flush and forget". I think the idea of passing a predicate to select series for finishing makes sense.

Sure, finish works.

Consumers should receive the correct finalized value of these series. As @ArthurSens points out in #4184, what we need is a specification for how the data should be transmitted so that the ending of a series is clear. In Prometheus, we have the NaN value, and in OTel we have the missing data point flag, but we've never specified how to set that flag. I would like to see a specification that dictates SDKs have to remember the "finishing" series long enough to send the NaN/missing-data-flag to each reader at least once, otherwise that reader would lose information. Then, to answer #4184, we need to specify that new series must be created with a start time >= the Nan/missing-data-flag previously issued for the same series.

Are you offering to work on that specification? Is it a requirement for this or a follow-up?

[atoulme]

This was discussed on 1/6/2026 as part of the maintainer sync meeting, I reproduce the meeting notes below:

• this PR is blocked on the SDK spec. The staleness marker discussion can be a separate PR.
• consider staleness marker from Prometheus, already exist in proto
  https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto#L375-L378

[dashpole]

The start time PR merged, so this should be unblocked now. I think there will be some updates to the start timestamp spec needed to clarify the interaction between finish and start times.

"All cumulative timeseries MUST use the initial start timestamp in subsequent collection intervals." probably needs to be changed to "All cumulative timeseries MUST use the initial start timestamp in subsequent collection intervals until Finish is called on the timeseries. After Finish is called, the next measurement to the timeseries SHOULD choose a new start time following the above rules for cumulative aggregations.".

[atoulme]

Thanks for the update @dashpole . I think we are unblocked but need to update the PR and the POCs. I have asked @MrAlias if he could help with the Go POC, and I'll try to refresh the Java one. It's been a bit and we need help ; the Python POC could be a nice one to update next.

• Review the language in the PR based on comments
• Reimplement/rebase Go POC
• Reimplement/rebase Java POC

[charless-splunk]

I also have a use case for being able to cleanly unregister metrics from the client at runtime...

We have multiple autoscaled services that primarily consumes from kafka. As scaling events happen, kafka partitions are reassigned. We scope many of our metrics to kafka partitions where partition is an important metadata tag on the metrics. We track things at a partition level like latest offset committed, records consumed, and some logical time tracking as based on specific messages consumed.

If we are unable to cleanly remove metrics at runtime, then we end up with long lived instances reporting stale values for a partition that was reassigned to another instance while that other instance reports current values for the given partition.

In the case of java applications, we risk a heap leak over time on the lowest order instances that live the longest and through many partition reassignments.

[dashpole]

Can you link the PoCs in the description?

[reyang]

"It will no longer be reported." - forever or until another API call trying to add the attribute set back?

[atoulme]

Registering again will create a new attribute set with a new start time.

[dashpole]

The instrument isn't identified by the attributes. I would consider "time series" or something similar. You also say this below. This also applies to other Finish function definition.

[dashpole]

Some things I would like to see specified in the SDK section:

• Specify how this interacts with export and storage. Something like "The SDK MUST preserve previously-aggregated metric data for the timeseries until it has been Collected, and then MUST clear all internal storage associated with the timeseries."
• Specify how this interacts with timestamps: Something like "When a an aggregation that has been Finished is Collected, its timestamp MUST be the time at which it was Finished". This may require changes elsewhere in the specification.
• Specify what happens if a timeseries has been finished, but a new measurement is made to it. This is a bit tricky. Options are:
  • [Preferred] Un-finish the timeseries. This is a new observation, so rather than stop + start again, we can just continue the existing aggregation.
  • Move the timestamp back to the current measurement time, but still Finish the timeseries. This might help if there is a race where a measurement is made just after it is Finished.
  • Keep the existing timeseries finished, but start a new one with a new start timestamp. This might be complex to implement, as they would both have the same attribute set.

[jack-berg]

[Preferred[ Un-finish the timeseries. This is a new observation, so rather than stop + start again, we can just continue the existing aggregation.

👍

[dashpole]

I think we should change this to be able to delete one or more timeseries by treating attributes that are not provided to this as matching. There are a few reasons:

• I can easily delete all timeseries on an instrument by providing no attributes
• As a user, I don't have to keep track of every attribute that i've ever provided to the API. For example, if my HTTP server dynamically creates and deletes routes over time, I would like to be able to Finish all timeseries associated with a particular http.route without needing to enumerate all of the status codes it has ever served.
• This would interact much better with filtering. If, for example, I was filtering out all attributes except for http.route on my server metrics, it isn't clear if the SDK should Finish the aggregation with http.route=foo if I call Finish(http.route=foo, http.response.status_code=200, ...). Was my intent to delete the entire route? Was it to only delete the portion that has status code 200? If, instead, users of the API pass attributes where they want all aggregations containing those attributes to be deleted, the previous example would not delete anything. If I instead called Finish(http.route=foo), only then would the entire route be deleted.

[jack-berg]

Yes. Why not the proposed predicate solution?

I can't find any comments which indicate there are problems / critiques with it.

[dashpole]

Ah, sorry. Missed it. I like your approach better, and support allowing users to provide a predicate or filter function.

I think the only downside is possibly performance, but I don't expect that to be a big concern for Finish.

[atoulme]

Simply out of bandwidth. I will work on it for java.

[MrAlias]

We updated the Go PoC to track the current Finish proposal and used it to explore a few API and SDK behaviors that are not yet fully spelled out in the spec text.

The current Go PoC now includes:

• Finish on synchronous instruments with a dedicated FinishOption surface
• exact attribute-set finishing, aligned with the current spec text
• SDK tombstoning semantics: a finished series is exported one final time and then dropped on the following cycle
• correct revival semantics: if a finished series is measured again before collection, the series is revived without dropping previously accumulated data
• fresh cumulative start times when a finished series is later recreated
• an additional PoC-only matcher-based finish option, WithMatchAttributes(func(attribute.Set) bool), to explore broader-than-exact matching

A few observations from the Go implementation:

1. Tombstoning appears important. Immediate deletion was too weak in practice. For a Finish operation to be observable and predictable, the SDK likely needs final export, then forget semantics rather than immediate disappearance.

2. Revival before collection is subtle. A naive tombstone implementation can lose data if a series is finished and then measured again before the next collection. In the PoC, we had to explicitly revive the tombstoned state rather than simply delete the tombstone marker.

3. Exact-match-only Finish may be too limiting. The current spec text is framed in terms of identifying an attribute set, and the Go PoC supports that directly. We also explored a matcher-based option because users may want to finish broader groups of series, for example:

   • all series for a container regardless of other attributes
   • all series with a specific attribute value
   • a narrower subset by combining exact attributes with a predicate

4. API shape matters. For Go, a dedicated FinishOption surface worked better than reusing measurement options. Finish is not a measurement operation, and keeping its option space separate made both the public API and SDK implementation cleaner.

Based on the PoC, I think the spec would benefit from being more explicit about:

• whether Finish is intended to be exact-match only, or whether broader matching should be supported
• what should happen if a finished series is measured again before collection
• whether a finished series should still be exported one final time
• cumulative start-time behavior when a finished series is later recreated

If useful, I can also summarize the concrete Go API/SDK tradeoffs we hit while implementing the PoC.

[jack-berg]

@atoulme joined the 5/14 java SIG yesterday to discuss this.

I asked some questions, following feedback on the java prototype:

• When should instrumentation call this?
• What is the consequence if they don't?
• What about series with attributes which are computed at record time, where the instrumentation has no idea when these series will no longer be used?

We need to be able to have answers to these types of questions to avoid bad / confusing UX. From my POV, the answers aren't good. If the application configured the SDK to use delta temporality, you don't have to worry about calling finish at all. If cumulative, it can help but only in situations where the series are known ahead of time and the caller is aware of their lifecycle. If a caller doesn't call finish, they get the behavior they've been getting the whole time: cumulative metrics stay in memory for the lifetime of the application.

We started talking about an alternative solution:

What if instead of a finish method, there was a view configuration option (or MeterProvider level, or Meter level, or MetricReader level) that allowed you to specify the TTL for stale cumulative series? I.e. "I want to retain cumulative series for 10 collection cycles (or time based) beyond their last measurement". And what if we coupled this with a new instrument advisory parameter which allowed instrumentation to inform the SDK about how series lifecycle works for that instrument: "for instrument, all series receive measurements regularly, so the absence of a measurement indicates that the series is stale and should be removed after the next collection"

This type of solution has a much better UX:

• When should instrumentation call this? They can set the advisory parameter all the time if they want.
• What is the consequence if they don't? The behavior might not be as good out of the box, but the application can still get the behavior they want by configuring the SDK.
• What about series with attributes which are computed at record time, where the instrumentation has no idea when these series will no longer be used? Instrumentation can set an advisory parameter that best reflects this. The application get the behavior they want by configuring the SDK.

Thoughts @dashpole, @MrAlias, @reyang, @jmacd?

[dashpole]

When should instrumentation call this?
What is the consequence if they don't?

Instrumentation should call this when one or more attribute values aren't expected to ever be recorded again. E.g. a pod name of a pod that was deleted, a database pool (from the original issue.

If it isn't removed, then cumulative metrics have a memory/timeseries leak. For delta metrics, some downstream components (e.g. deltatocumulative processor) won't know if a timeseries should still be considered active.

What about series with attributes which are computed at record time, where the instrumentation has no idea when these series will no longer be used?

Finish is only really useful when the label you care about is not dynamic. With the database pool example above, you would need to know the database pool name, but don't need to know all of the dynamic database pool statuses (made up dynamic attribute). I think we had already discussed a predicate-based API.

What about series with attributes which are computed at record time, where the instrumentation has no idea when these series will no longer be used? Instrumentation can set an advisory parameter that best reflects this. The application get the behavior they want by configuring the SDK.

Can you give an example? In some sense, an http client has no idea when you will get a 429 again, but I wouldn't want that series cleaned up (I think it would disappear from my graphs instead of showing a rate of zero).

I think there are additional ways Finish would be useful. One minor thing is that you would get more accurate end timestamps when something you were monitoring (e.g. the pod or database pool) is deleted. Another is staleness markers (NoDataPointFlag) so that UIs know where to stop drawing lines. We had also discussed this being useful in the context of Bound instruments.

If we wanted to go with an expiration to solve this use-case, I would ideally be able to set a TTL on a list of attribute keys. E.g. For each value of database.pool.name and production.env, if no observations are made about that a particular pool in a few collection intervals, then delete all timeseries with that database.pool.name and production.env. But tuning the interval might be hard for rarely-observed metrics. You would basically be converting the attributes you would track for the Finish call into an advisory parameter, and having the SDK do the work. You wouldn't get as precise of end times, though.

[jack-berg]

If it isn't removed, then cumulative metrics have a memory/timeseries leak.

They already leak today. If http.server.request.duration is using cumulative temporality, it may record to series representing a rare combination of attributes and never record to it again. And this finish method won't solve that. That's the problem I see: finish makes a problem that only exists in certain SDK configurations (i.e. cumulative) a instrumentation concern, and it only solves the problem in certain situations.

Can you give an example? In some sense, an http client has no idea when you will get a 429 again, but I wouldn't want that series cleaned up (I think it would disappear from my graphs instead of showing a rate of zero).

I prefer to think about this from the perspective of the metrics SDK arather than the graph, because graph / query behavior varies by platform. Consider the http.server.request.duration metric with a rare combination of attributes I mentioned earlier: Since this hypothetical series receives measurements rarely, a newly started application might not record to that series for days / months / ever. Once it does receive a rare measurement, it might not receive any more. Why should it need to keep emitting the it for days / weeks / etc, and how would imposing a TTL where it stops after some period be different than the original state for the days / months leading up to the first measurement?

One minor thing is that you would get more accurate end timestamps when something you were monitoring (e.g. the pod or database pool) is deleted.

Yes it would allow for an explicit end timestamp corresponding to the moment when finish was called. That's not possible with a TTL based approach.

If we wanted to go with an expiration to solve this use-case, I would ideally be able to set a TTL on a list of attribute keys.

That seems complicated to implement and for the user to reason about for benefit which I think would be pretty niche.

[dashpole]

They already leak today. If http.server.request.duration is using cumulative temporality, it may record to series representing a rare combination of attributes and never record to it again. And this finish method won't solve that.

I wouldn't call this a leak, and it sounds like we might differ on the problem we are trying to solve.

In K8s SIG instrumentation, when defining new metrics for k8s components, we don't have any concerns about rare status codes because they are bounded. We don't consider it a leak to keep all status codes around, because it has to keep leaking for a long time before it causes problems. What we do worry about is an attribute where the cardinality of the values isn't bounded, and where the cardinality will continue to grow over the lifetime of the process. As far as i'm aware, this has always been when an attribute is a reference to something external (a pod, a volume, service etc), and those things come and go over time. In those cases, we use an API that allows explicitly managing the state for metrics ourselves.

Monitoring external resources is a niche pattern (typically done via the collector in OTel today), and nearly all instrumentation libraries won't need to use Finish.

It would maybe also be nice to have a TTL to also intelligently drop series you don't really need. But the thing that is missing from OTel today is that you can't use it to monitor a set of things that change over time.

a newly started application might not record to that series for days / months / ever

Fair point; you are right. I agree cleaning up a metric probably shouldn't be a concern. It will probably surprise some users, but I don't think anything will break.

[jmacd]

Based on other's feedback above, it seems to me that we can have what we want without a new API surface.

an http client has no idea when you will get a 429 again, but I wouldn't want that series cleaned up (I think it would disappear from my graphs instead of showing a rate of zero).

I would say that that what you do as a receiver of the information is independent of what the producer decides to do. If you haven't seen any 429 observations in a while, you should be free to issue a final point with the NO_DATA flag set. When you receive that there is no data in a timeseries, you are free to create a gap in your series or to fill in zeros.

I think we should have automatic TTLs; any SDK that decides to remove from its memory and stop reporting a timeseries should issue a final point with NO_DATA set. SDKs that begin reporting the same series again must use a new start timestamp. Why do we need an explicit API, especially given the problems pointed at, e.g.,

What about series with attributes which are computed at record time, where the instrumentation has no idea when these series will no longer be used?

This is the norm for a statsd/dogstatsd API user. Delta temporality is the solution for this use-case, and SDKs in my opinion should be organized not to accumulate memory under these conditions. In a related note, I'm aware of large users who have been unable to migrate to OTel away from statsd/dogstatsd because those APIs let users define metric instruments "at record time"; and likewise, I think OTel SDKs should be able to efficiently handle instruments being created on-demand.