Prometheus protocols and PromQL

Expose ClickHouse server metrics

If you’re using ClickHouse Cloud, you can expose metrics to Prometheus using the Prometheus Integration.

Configure a dedicated port when a Prometheus server needs to scrape ClickHouse’s own metrics:

<prometheus>
    <port>9363</port>
    <endpoint>/metrics</endpoint>
    <metrics>true</metrics>
    <asynchronous_metrics>true</asynchronous_metrics>
    <events>true</events>
    <errors>true</errors>
    <histograms>true</histograms>
    <dimensional_metrics>true</dimensional_metrics>
</prometheus>

Section <prometheus.handlers> can be used to make more extended handlers on the same port. This section is similar to <http_handlers> but works for prometheus protocols:

<prometheus>
    <port>9363</port>
    <handlers>
        <my_rule_1>
            <url>/metrics</url>
            <handler>
                <type>expose_metrics</type>
                <metrics>true</metrics>
                <asynchronous_metrics>true</asynchronous_metrics>
                <events>true</events>
                <errors>true</errors>
                <histograms>true</histograms>
                <dimensional_metrics>true</dimensional_metrics>
                <labels>
                    <environment>production</environment>
                    <shard from_env="SHARD_NAME"></shard>
                </labels>
            </handler>
        </my_rule_1>
    </handlers>
</prometheus>

Settings:

Name	Default	Description
`port`	none	Port that serves ClickHouse metrics.
`endpoint`	`/metrics`	HTTP endpoint for scraping metrics. It starts with `/`. Should not be used with the `<handlers>` section.
`url` / `headers` / `method`	none	Filters used to find a matching handler for a request. Similar to the fields with the same names in the `<http_handlers>` section.
`info`	true	Exposes the `ClickHouse_Info` gauge with server identity labels (`name`, `version`, `version_describe`, `version_major`, `version_minor`, `version_patch`).
`metrics`	true	Exposes metrics from `system.metrics`.
`asynchronous_metrics`	true	Exposes metrics from `system.asynchronous_metrics`.
`events`	true	Exposes metrics from `system.events`.
`errors`	true	Exposes error counts from `system.errors`.
`histograms`	true	Exposes metrics from `system.histogram_metrics`.
`dimensional_metrics`	true	Exposes metrics from `system.dimensional_metrics`.
`labels`	none	Constant labels added to every exposed metric. Each child element defines one label: the element name is the label name (which must match `[a-zA-Z_][a-zA-Z0-9_]*`) and the element value is the label value. Label values support standard config substitutions such as the `from_env` attribute. A label name is rejected when it starts with `__` (reserved by Prometheus), or when it would collide with a label this endpoint already writes for one of its enabled sections. The reserved set therefore follows the endpoint’s active export surface: `le` when `histograms` is enabled; the `ClickHouse_Info` labels (`name`, `version`, `version_describe`, `version_major`, `version_minor`, `version_patch`) when `info` is enabled; and any label used by an exposed histogram or dimensional metric family (for example, `group`, `direction`, or `operation_type`) when `histograms` or `dimensional_metrics` is enabled. Because it depends on what the endpoint actually exposes, a name can be valid on one endpoint but rejected on another.

Check the endpoint:

curl http://127.0.0.1:9363/metrics

Prometheus HTTP API and PromQL

ClickHouse implements the Prometheus HTTP API over a TimeSeries table. One handler serves remote write, remote read, instant PromQL queries, and range PromQL queries.

Prerequisites

Enable the allow_experimental_time_series_table setting for the user that creates and accesses the table:

SET allow_experimental_time_series_table = 1;

Create a database and a TimeSeries table:

CREATE DATABASE prometheus;
CREATE TABLE prometheus.metrics ENGINE = TimeSeries;

For HTTP API requests, enable allow_experimental_time_series_table in the profile of the API user.

Configure the Prometheus API

Configure one prefix-routed handler on the main ClickHouse HTTP port:

<http_handlers>
    <defaults/>
    <rule>
        <url_prefix>/prometheus/api/v1</url_prefix>
        <handler>
            <type>prometheus_api_v1</type>
        </handler>
    </rule>
</http_handlers>

<defaults/> preserves the built-in handlers for endpoints such as /ping and for SQL requests. The prefix above exposes these endpoints through one handler:

Endpoint	Purpose
`/prometheus/api/v1/write`	Prometheus remote write
`/prometheus/api/v1/read`	Prometheus remote read
`/prometheus/api/v1/query`	Instant PromQL queries
`/prometheus/api/v1/query_range`	Range PromQL queries
`/prometheus/api/v1/format_query`	PromQL expression formatting
`/prometheus/api/v1/series`	Series metadata
`/prometheus/api/v1/metadata`	Metric-family metadata

The example omits database and table from the handler. Each request must provide the table query parameter (except for /format_query, which only parses the given PromQL expression and doesn’t need a table). It can also provide database, use a qualified table name such as prometheus.metrics, or omit the database to use default. This allows one handler to serve multiple TimeSeries tables. To use one fixed table for every request, configure it in the handler:

<handler>
    <type>prometheus_api_v1</type>
    <database>prometheus</database>
    <table>metrics</table>
</handler>

A table configured in the handler cannot be overridden by request parameters. Routing and handler settings:

Name	Default	Description
`url_prefix`	none	Rule filter that matches every request path that starts with the configured prefix.
`table`	none	The name of a `TimeSeries` table. When omitted, the request must provide the `table` query parameter. The configured name can include a database.
`database`	none	The database containing the table. A request can provide it as a query parameter. When omitted, ClickHouse uses a database from a qualified `table` value or falls back to `default`.

Ingest metrics with remote write

ClickHouse supports the Prometheus remote-write protocol. Configure Prometheus to write to the handler:

remote_write:
  - url: https://clickhouse.example.com:8443/prometheus/api/v1/write?database=prometheus&table=metrics
    basic_auth:
      username: default
      password: <password>

Prometheus sends samples to the prometheus.metrics table. To batch data from many concurrent remote-write requests into fewer parts, enable asynchronous inserts by adding the async_insert setting to the URL (or by enabling it in the user profile):

remote_write:
  - url: https://clickhouse.example.com:8443/prometheus/api/v1/write?database=prometheus&table=metrics&async_insert=1

ClickHouse acknowledges an asynchronous remote-write request only after the data is flushed to all inner tables of the TimeSeries table, regardless of the wait_for_async_insert setting: the remote-write protocol treats an acknowledged write as durable. If the flush fails, the request returns an error and Prometheus retries it.

Query with PromQL

Use the instant-query endpoint to evaluate a PromQL expression at one point in time:

curl --user default:<password> --get \
  "https://clickhouse.example.com:8443/prometheus/api/v1/query" \
  --data-urlencode "query=rate(http_requests_total[5m])" \
  --data-urlencode "database=prometheus" \
  --data-urlencode "table=metrics"

Use the range-query endpoint to evaluate an expression over a time range:

curl --user default:<password> --get \
  "https://clickhouse.example.com:8443/prometheus/api/v1/query_range" \
  --data-urlencode "query=rate(http_requests_total[5m])" \
  --data-urlencode "start=2026-08-15T12:00:00Z" \
  --data-urlencode "end=2026-08-15T13:00:00Z" \
  --data-urlencode "step=60s" \
  --data-urlencode "database=prometheus" \
  --data-urlencode "table=metrics"

Use the format-query endpoint to parse and format a PromQL expression without evaluating it:

curl --user default:<password> --get \
  "https://clickhouse.example.com:8443/prometheus/api/v1/format_query" \
  --data-urlencode "query=sum by(job)(http_requests_total{code=\"200\"})/2"

The expression is returned serialized from the parsed query, with the whitespace normalized, the comments removed, the redundant parentheses dropped, and the durations converted to numbers of seconds: sum by (job) (http_requests_total{code="200"}) / 2. This endpoint doesn’t evaluate the expression, so it doesn’t need the database and table parameters. See the supported PromQL features for the function and aggregation operator list used by the HTTP API, the promql dialect, and the table functions.

Grafana

Configure a Prometheus data source with the base URL ending before /api/v1:

apiVersion: 1
datasources:
  - name: ClickHouse Prometheus
    type: prometheus
    access: proxy
    url: https://clickhouse.example.com:8443/prometheus
    basicAuth: true
    basicAuthUser: default
    jsonData:
      httpMethod: GET
      customQueryParameters: database=prometheus&table=metrics
    secureJsonData:
      basicAuthPassword: <password>

Grafana appends /api/v1/query or /api/v1/query_range to this base URL and adds customQueryParameters to each request.

Only the query endpoints /api/v1/query, /api/v1/query_range, and /api/v1/format_query and the metadata endpoints /api/v1/series, /api/v1/labels, /api/v1/label/<name>/values, and /api/v1/metadata are implemented. /api/v1/series requires at least one match[] series selector, supports the optional start, end, and limit parameters, and returns the union of the series matched by each selector. /api/v1/labels accepts the same parameters, with match[] being optional, and returns the sorted label names of the matched series (or of all series when no selectors are given). /api/v1/label/<name>/values accepts the same parameters as /api/v1/labels and returns the sorted values of one label, with <name> optionally using the Prometheus U__... escaping of label names that contain characters outside [a-zA-Z0-9_]. These endpoints cover what a Grafana Prometheus datasource uses for label browsing, template variables, and query-builder autocomplete.

SQL entry points

ClickHouse uses the same PromQL converter for the HTTP API, the promql dialect, and the prometheusQuery and prometheusQueryRange table functions. Run PromQL directly with clickhouse-client:

clickhouse-client \
  --dialect promql \
  --promql_database prometheus \
  --promql_table metrics \
  --query 'rate(http_requests_total[5m])'

Use the table functions to embed PromQL in a SQL query:

SELECT *
FROM prometheusQuery(
    prometheus.metrics,
    'rate(http_requests_total[5m])',
    now()
);

Query metric metadata

The /prometheus/api/v1/metadata endpoint returns the metric metadata stored in the Metrics target table of the TimeSeries table: the type, help text, and unit of each metric family. It supports the following Prometheus parameters in the URL query string:

Parameter	Description
`metric`	Return metadata only for this metric family.
`limit`	Limit the number of returned metric families. A negative value means no limit; zero returns no metric families.
`limit_per_metric`	Limit the number of metadata objects returned for each metric family. Zero and negative values mean no limit.

The default Metrics target table is a ReplacingMergeTree ordered by the metric family name: it keeps the most recently written metadata entry for each metric family. Several entries per family are returned only while the target table stores them — before its parts are merged, or when the table is defined with an engine that preserves them.

curl --user default:<password> --get \
  "https://clickhouse.example.com:8443/prometheus/api/v1/metadata" \
  --data-urlencode "metric=http_requests_total" \
  --data-urlencode "database=prometheus" \
  --data-urlencode "table=metrics"

Read metrics with remote read

ClickHouse supports the Prometheus remote-read protocol at /prometheus/api/v1/read. Configure a Prometheus server to read from the same TimeSeries table:

remote_read:
  - url: https://clickhouse.example.com:8443/prometheus/api/v1/read?database=prometheus&table=metrics
    basic_auth:
      username: default
      password: <password>

Fundamentals

ClickHouse concepts

Best practices

Expose ClickHouse server metrics

Prometheus HTTP API and PromQL

Prerequisites

Configure the Prometheus API

Ingest metrics with remote write

Query with PromQL

Grafana

SQL entry points

Query metric metadata

Read metrics with remote read

​Expose ClickHouse server metrics

​Prometheus HTTP API and PromQL

​Prerequisites

​Configure the Prometheus API

​Ingest metrics with remote write

​Query with PromQL

​Grafana

​SQL entry points

​Query metric metadata

​Read metrics with remote read

Expose ClickHouse server metrics

Prometheus HTTP API and PromQL

Prerequisites

Configure the Prometheus API

Ingest metrics with remote write

Query with PromQL

Grafana

SQL entry points

Query metric metadata

Read metrics with remote read