Helper functions for PromQL-like time series data queries by davenger · Pull Request #80590 · ClickHouse/ClickHouse

davenger · 2025-05-20T17:36:21Z

Aggregate functions
timeSeriesResampleToGridWithStaleness - resample time series data to grid with specified step
timeSeriesDeltaToGrid - calculate Prometheus delta for a time series on grid with specified step
timeSeriesRateToGrid - calculate Prometheus rate for a time series on grid with specified step
timeSeriesInstantDeltaToGrid - calculate Prometheus idelta for a time series on grid with specified step
timeSeriesInstantRateToGrid - calculate Prometheus irate for a time series on grid with specified step

All these functions accept 2 parameters: timestamp and value and return Array(Nullable(Float64))

Example query to resample time series data to grid starting at ts=90 ending at ts=90+120 with step 15: ([90, 105, 120, 135, 150, ... , 210])

WITH
    [110, 120, 130, 140, 190, 200, 210, 220, 230] AS timestamps, -- note: the gap between 140 and 190 is to show how values are filled for ts = 150, 165, 180 
    [1, 1, 3, 4, 5, 5, 8, 12, 13] AS values, -- array of values corresponding to timestamps above
    90 AS start_ts,       -- start of timestamp grid
    90 + 120 AS end_ts,   -- end of timestamp grid
    15 AS step_seconds,   -- step of timestamp grid
    30 AS window_seconds  -- "staleness" window
SELECT timeSeriesResampleToGridWithStaleness(start_ts, end_ts, step_seconds, window_seconds)(ts, value)
FROM
(   -- This subquery just converts array of timestamps and values into rows of (ts, value) 
    SELECT
        arrayJoin(arrayZip(timestamps, values)) AS ts_and_val,
        ts_and_val.1::DateTime AS ts,
        ts_and_val.2::Float64 AS value
)
SETTINGS allow_experimental_ts_to_grid_aggregate_function=1;

   ┌─timeSeriesResa⋯s)(ts, value)─┐
1. │ [NULL,NULL,1,3,4,4,NULL,5,8] │
   └──────────────────────────────┘

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

timeSeries* helper functions to speedup some scenarios when working with time series data:

re-sample the data to the time grid with specified start timestamp, end timestamp and step
calculate PromQL-like delta, rate, idelta and irate.

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

clickhouse-gh · 2025-05-20T17:36:47Z

Workflow [PR], commit [ba3bc4e]

…etate functions

alexey-milovidov · 2025-05-21T15:52:42Z

How about renaming Idelta to IDelta and Irate to IRate?
By the way, what is I? Maybe expand it?

…etate functions

vitlibar · 2025-05-22T13:58:12Z

How about renaming Idelta to IDelta and Irate to IRate? By the way, what is I? Maybe expand it?

It's named idelta() in PromQL. There are two similar functions in PromQL: delta() and idelta(). I haven't found any information about what exactly that letter I means. But it's definitely a prefix.

davenger · 2025-05-22T14:10:08Z

By the way, what is I?

AFAIU "I" stands for "instant". irate and idelta calculate instant values based on latest 2 samples while rate and delta calculate their values based on oldest and newest samples (actually they also take into account other samples in-between) within the specified window.

vitlibar · 2025-05-22T16:10:32Z

+                }
+                else if (timestamp == timestamps[0])
+                {
+                    /// Replace the value with bigger one


Why do we replace the value with bigger one?

This is done for stability. Normally, there should not be samples with equal timestamps but if for some reason they are present in user data we want to always choose the same one no mater in which order the data is scanned.

vitlibar · 2025-05-22T16:12:25Z

+    struct Data
+    {
+        ValueType values[2];            /// In common scenario values are Float64, so put them first as they need 8-byte alignment
+        TimestampType timestamps[2];    /// Timestamps might be stored as DateTime64, DateTime32 or even as 16-bit delta from the base timestamp of the grid


Is it always true that timestamps[2] < timestamps[1] < timestamps[0]?

There can't be timestamp[2], as we store no more than 2 samples and yes, timestamps[0] is the latest

vitlibar · 2025-05-22T20:17:06Z

+[(1734955380,NULL),(1734955395,NULL),(1734955410,NULL),(1734955425,0),(1734955440,0),(1734955455,0),(1734955470,0),(1734955485,0),(1734955500,0),(1734955515,1),(1734955530,3),(1734955545,5),(1734955560,5),(1734955575,5),(1734955590,5),(1734955605,8),(1734955620,8),(1734955635,8),(1734955650,8),(1734955665,8),(1734955680,8)]
+Row 1:
+──────
+rate_5m:  [(1734955380,NULL),(1734955395,NULL),(1734955410,NULL),(1734955425,NULL),(1734955440,0),(1734955455,0),(1734955470,0),(1734955485,0),(1734955500,0),(1734955515,0.003467629629629629),(1734955530,0.010345333333333333),(1734955545,0.017170277777777774),(1734955560,0.017114320987654322),(1734955575,0.017069555555555557),(1734955590,0.01703292929292929),(1734955605,0.027203851851851854),(1734955620,0.02716252991452992),(1734955635,0.027127111111111112),(1734955650,0.02709641481481482),(1734955665,0.02706955555555555),(1734955680,0.02704585620915033)]


I tried to check value (1734955515,0.003467629629629629) here.
The window was 300 s, so the first point in the window was (1734955421.374,0) and the last point was (1734955511.374,1). Thus it seems the rate should be (1 - 0) / (1734955511.374 - 1734955421.374) which is 0.011111111 and not 0.003467629629629629. Did I miss something?

Prometheus also does some extrapolation. I actually copied the code for rate and delta calculation from Prometheus to make those computation fully compatible. See here https://github.com/ClickHouse/ClickHouse/pull/80590/files#diff-c2739d27c54dfe109d9cf36ed500c9ba18fa1d8eaadc3a7be59558676e64cc7aR147-R198

Yes, I see it now. It indeed used extrapolation:

range 1734955215..1734955515 (300s): range_start = 1734955215 range_end = 1734955515 t1 = 1734955421.374 v1 = 0 t2 = 1734955511.374 v2 = 1 average_interval = 15 (average time difference between points) raw_increase = (v2 - v1) = (1 - 0) = 1 raw_rate = raw_increase / (t2 - t1) = 1 / (1734955511.374 - 1734955421.374) = 0.011111111 post_interpolation = (range_end - t2) * raw_rate = (1734955515 - 1734955511.374) * 0.011111111 = 0.040288888 pre_interpolation = 0 (because v1 = 0 and interpolation doesn't go to negative values) increase = raw_increase + pre_interpolation + post_interpolation = 1 + 0 + 0.040288888 = 1.040288888 rate = increase / (range_end - range_start) = 1.040288888 / 300 = 0.00346763

vitlibar · 2025-05-22T20:23:00Z

+(1734955646.374, 8),
+(1734955661.374, 8),
+(1734955676.374, 8)
+]);


This time series doesn't contain resets. Did you manage to take resets into account when calculating rate() or delta()?

This test data has resets

ClickHouse/tests/queries/0_stateless/03254_timeseries_instant_value_aggregate_functions.sql

Lines 17 to 19 in ae39773

(10, 42, '2024-12-12 12:00:20', (['2024-12-12 12:00:19'], [110])),

(10, 42, '2024-12-12 12:00:30', (['2024-12-12 12:00:29', '2024-12-12 12:00:23'], [100, 100])),

(10, 42, '2024-12-12 12:00:40', (['2024-12-12 12:00:39', '2024-12-12 12:00:38'], [90, 100]));

And here rate and delta are calculated

ClickHouse/tests/queries/0_stateless/03254_timeseries_instant_value_aggregate_functions.sql

Lines 34 to 36 in ae39773

timeSeriesRateToGrid(start_ts, end_ts, step_sec, window_sec)(samples.1, samples.2) as rate_values,

timeSeriesRateToGrid(start_ts, end_ts, step_sec, window_sec)(samples.1::Array(DateTime64(5, 'UTC')), samples.2) as rate_values_scale_5,

timeSeriesDeltaToGrid(start_ts, end_ts, step_sec, window_sec)(samples.1, samples.2) as delta_values

vitlibar · 2025-05-22T20:41:15Z

+
+void registerAggregateFunctionTimeseriesToGrid(AggregateFunctionFactory & factory)
+{
+    factory.registerFunction("tsToGrid", createAggregateFunctionTimeseriesToGrid);


Function tsToGrid() isn't mentioned in the PR's description.

Removed tsToGrid in favor of timeSeriesResampleToGridWithStaleness

davenger · 2025-05-26T12:17:43Z

By the way, what is I? Maybe expand it?

Renamed timeSeriesIrateToGrid - > timeSeriesInstantRateToGrid, timeSeriesIdeltaToGrid - > timeSeriesInstantDeltaToGrid

Co-authored-by: Alon Tal <alon@alontal.com>

vitlibar · 2025-06-04T15:50:02Z

+
+/// Aggregate function to store last (latest by timestamp) 2 samples with distinct timestamps
+template <typename TimestampType, typename ValueType>
+class AggregateFunctionLast2Samples final :


Is this aggregation function used anywhere except that its Data is used to implement timeSeriesInstantRateToGrid()?

Almost forgot about it.
This is a helper function that is useful for building MV and table with re-sampled data. It helps to reduce the amount of data that has to be read and processed for queries with grid timestamp step much bigger then raw data time resolution.

Added it to the docs:
https://github.com/ClickHouse/ClickHouse/pull/80590/files#diff-ddd55e57d7c3280a3a719f9939480a1971efe8cb51c6200468fc708bdcfa7a21

vitlibar · 2025-06-04T15:51:50Z

+        }
+    }
+
+    static constexpr UInt16 FORMAT_VERSION = 2;


When was FORMAT_VERSION equal to 1?

It was used during testing with an early adopter customer, so for compatibility it makes sense to not reset it.

vitlibar · 2025-06-04T15:55:16Z

+namespace DB
+{
+
+template <bool array_agruments_, typename TimestampType_, typename IntervalType_, typename ValueType_, bool is_rate_>


What does array_arguments_ mean? Which test checks such a case?

timestamp an value parameters of all these functions can be either individual values or arrays. It's described in the docs:
https://github.com/ClickHouse/ClickHouse/pull/80590/files#diff-ede8b2e1838fa4dc7688a22667c5cf49988c061d62c0d81ce4d9f1be21e82e83R54-R64

array_arguments_ template parameter switches the implementation to array params.

Many tests pass timestamps and values as array. Here are some of them:
https://github.com/ClickHouse/ClickHouse/pull/80590/files#diff-e00e69552b275c579e7cd6ff9fde81f5963eb8c0003b9692777b2f63d22bd055R1-R14

…FieldBinary

davenger · 2025-06-05T18:08:30Z

CH Inc sync — tests failed - test 03145_non_loaded_projection_backup is flaky in private repo

Performance Comparison (amd_release,master_head,1/3)
Performance Comparison (amd_release,master_head,1/3) — 3 faster, 8 slower, 24 unstable
Performance Comparison (amd_release,master_head,2/3)
Performance Comparison (amd_release,master_head,2/3) — 5 faster, 6 slower, 14 unstable

performance runs look unstable

thevar1able · 2025-06-06T23:30:52Z

#81399

clickgapai · 2026-04-07T08:22:43Z

Hi @davenger @vitlibar — while reviewing this PR I found the following:

Bug (P1): Decimal32 parameter normalization truncates due to divide-before-multiply order — Decimal32 parameter normalization truncates due to divide-before-multiply order #101910

Happy to discuss — close anything that's wrong or already addressed.

Helper functions for PromQL-like time series data queries

d4521a5

davenger marked this pull request as draft May 20, 2025 17:36

clickhouse-gh Bot added the pr-feature Pull request with new product feature label May 20, 2025

vitlibar self-assigned this May 20, 2025

davenger added 4 commits May 20, 2025 20:45

Fix for idelta with metric decrease

3abdaf7

Fix test with non-Atomic database

c397ba8

Tests for mismatching parameter values

3cb8aed

Pass parameter values to prevent combining mismatching states of aggr…

b139f99

…etate functions

davenger added 2 commits May 22, 2025 08:52

Pass parameter values to prevent combining mismatching states of aggr…

2acb0c1

…etate functions

Force enable_analyzer for proper column names

e81a1f5

vitlibar reviewed May 22, 2025

View reviewed changes

davenger added 2 commits May 23, 2025 20:06

Remove tsToGrid in favor of timeSeriesResampleToGridWithStaleness

c8a96ae

Renamed Irate and Idelta to InstantRate and InstantDelta

ae39773

Docs for timeSeriesResampleToGridWithStaleness

6413d61

davenger force-pushed the timeseries_aggregate_functions branch from d969ca0 to 6413d61 Compare May 30, 2025 11:19

davenger added 2 commits May 30, 2025 14:10

Style cleanups

ccebbd3

Docs to rate and delta time series functions

063a53b

davenger marked this pull request as ready for review May 30, 2025 13:25

Added idelat and irate

de42996

alont suggested changes May 30, 2025

View reviewed changes

Fix grammar

d452b33

Co-authored-by: Alon Tal <alon@alontal.com>

davenger added 3 commits June 3, 2025 13:58

Add allow_experimental_time_series_aggregate_functions

5e0c5c5

Clarify docs around NULL return values

5e107fd

Renamed window parameter

e8795c9

vitlibar approved these changes Jun 4, 2025

View reviewed changes

vitlibar reviewed Jun 4, 2025

View reviewed changes

davenger added 11 commits June 4, 2025 19:25

Tests If-combinator with Nullable conditions and array params

03d8adb

Fix for Nullable condition passed to If-combinator

7647e40

Enable analyzer for remote query

0fa689e

Address potential int overflow

d95b051

Merge branch 'master' into timeseries_aggregate_functions

b9c1de3

Add missing readFieldBinary for decimal types to match existing write…

7385d46

…FieldBinary

Another test for timeSeriesLastTwoSamples

3b258e2

Docs for timeSeriesLastTwoSamples

68689a8

Added timeSeriesLastTwoSamples

ddb1e26

Fix typo

87f45ec

Removed tabs

ba3bc4e

davenger added this pull request to the merge queue Jun 5, 2025

Merged via the queue into master with commit d0326a1 Jun 5, 2025
117 of 122 checks passed

davenger deleted the timeseries_aggregate_functions branch June 5, 2025 18:28

robot-ch-test-poll added the pr-synced-to-cloud The PR is synced to the cloud repo label Jun 5, 2025

alexey-milovidov mentioned this pull request Jun 7, 2025

Revert "Helper functions for PromQL-like time series data queries" #81462

Merged

davenger restored the timeseries_aggregate_functions branch June 10, 2025 20:33

davenger mentioned this pull request Jun 11, 2025

Revert "Revert "Helper functions for PromQL-like time series data queries"" #81627

Merged

1 task

clickgapai mentioned this pull request Apr 7, 2026

Decimal32 parameter normalization truncates due to divide-before-multiply order #101910

Open

	(10, 42, '2024-12-12 12:00:20', (['2024-12-12 12:00:19'], [110])),
	(10, 42, '2024-12-12 12:00:30', (['2024-12-12 12:00:29', '2024-12-12 12:00:23'], [100, 100])),
	(10, 42, '2024-12-12 12:00:40', (['2024-12-12 12:00:39', '2024-12-12 12:00:38'], [90, 100]));

	timeSeriesRateToGrid(start_ts, end_ts, step_sec, window_sec)(samples.1, samples.2) as rate_values,
	timeSeriesRateToGrid(start_ts, end_ts, step_sec, window_sec)(samples.1::Array(DateTime64(5, 'UTC')), samples.2) as rate_values_scale_5,
	timeSeriesDeltaToGrid(start_ts, end_ts, step_sec, window_sec)(samples.1, samples.2) as delta_values

Uh oh!

Conversation

davenger commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

Uh oh!

clickhouse-gh Bot commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexey-milovidov commented May 21, 2025

Uh oh!

vitlibar commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davenger commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vitlibar May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vitlibar May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davenger May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davenger commented May 26, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vitlibar Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davenger commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

thevar1able commented Jun 6, 2025

Uh oh!

davenger commented May 20, 2025 •

edited

Loading

clickhouse-gh Bot commented May 20, 2025 •

edited

Loading

vitlibar commented May 22, 2025 •

edited

Loading

davenger commented May 22, 2025 •

edited

Loading

vitlibar May 22, 2025 •

edited

Loading

vitlibar May 22, 2025 •

edited

Loading

davenger May 26, 2025 •

edited

Loading

vitlibar Jun 4, 2025 •

edited

Loading

davenger commented Jun 5, 2025 •

edited

Loading