Skip to content

Commit ab450f2

Browse files
317brian317brian
authored
docs: add docs for projections (#18056)
1 parent 803a231 commit ab450f2

File tree

3 files changed

+330
-0
lines changed

3 files changed

+330
-0
lines changed

‎docs/ingestion/ingestion-spec.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -396,6 +396,50 @@ The `filter` conditionally filters input rows during ingestion. Only rows that p
396396
ingested. Any of Druid's standard [query filters](../querying/filters.md) can be used. Note that within a
397397
`transformSpec`, the `transforms` are applied before the `filter`, so the filter can refer to a transform.
398398

399+
### Projections
400+
401+
Projections are ingestion/compaction time aggregations that Druid computes on a subset of dimensions and metrics of a segment. They are stored within a segment. The pre-aggregated data reduces the number of rows the query engine needs to process when you run a query. This can speed up queries for query shapes that match a projection.
402+
403+
Define projections for a new data source in the `projectionsSpec` block during ingestion. To add projections to an existing data source, see [create them afterwards](../querying/projections.md#manually-add-a-projection).
404+
405+
:::info
406+
Projections you define become a dimension for your datasource. To remove a projection from your datasource, you need to reingest the data with the projection removed. Alternatively, you can use a query context parameter to not use projections for a specific query.
407+
:::
408+
409+
```json
410+
"projectionsSpec": {
411+
"projections": [
412+
{
413+
"name": "daily_channel_summary",
414+
"dimensions": [
415+
"channel"
416+
],
417+
"granularity": "DAY",
418+
"metrics": [
419+
{
420+
"type": "longSum",
421+
"name": "total_added",
422+
"fieldName": "added"
423+
},
424+
{ "type": "longSum",
425+
"name": "total_deleted",
426+
"fieldName": "deleted"
427+
},
428+
{ "type": "longSum",
429+
"name": "total_delta",
430+
"fieldName": "delta"
431+
},
432+
{
433+
"type": "cardinality",
434+
"name": "distinct_users",
435+
"fieldName": "user"
436+
}
437+
]
438+
}
439+
]
440+
}
441+
```
442+
399443
### Legacy `dataSchema` spec
400444

401445
:::info

‎docs/querying/projections.md‎

Lines changed: 285 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,285 @@
1+
---
2+
id: projections
3+
title: Query projections
4+
sidebar_label: Projections
5+
description: Speed up your queries by defining projections that pre-aggreate data for you.
6+
---
7+
8+
import Tabs from '@theme/Tabs';
9+
import TabItem from '@theme/TabItem';
10+
11+
<!--
12+
~ Licensed to the Apache Software Foundation (ASF) under one
13+
~ or more contributor license agreements. See the NOTICE file
14+
~ distributed with this work for additional information
15+
~ regarding copyright ownership. The ASF licenses this file
16+
~ to you under the Apache License, Version 2.0 (the
17+
~ "License"); you may not use this file except in compliance
18+
~ with the License. You may obtain a copy of the License at
19+
~
20+
~ http://www.apache.org/licenses/LICENSE-2.0
21+
~
22+
~ Unless required by applicable law or agreed to in writing,
23+
~ software distributed under the License is distributed on an
24+
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
25+
~ KIND, either express or implied. See the License for the
26+
~ specific language governing permissions and limitations
27+
~ under the License.
28+
-->
29+
30+
:::info[Experimental]
31+
32+
Projections are experimental. We don't recommend them for production use.
33+
34+
:::
35+
36+
Projections are a type of aggregation that is computed and stored as part of your datasource in a segment. When using rollups to pre-aggregate rows are based on a specific granularity, the source rows are no longer available. Projections, on the other hand, don't affect the source dimensions. They remain part of your datasource and are queryable.
37+
38+
The pre-aggregated data can speed up queries by reducing the number of rows that need to be processed for any shape that matches a projection. Thus, we recommend you build projections for commonly used queries. For example, you define the following projection in your datasource:
39+
40+
41+
<details>
42+
<summary>Show the projection</summary>
43+
44+
```json
45+
46+
"projections": [
47+
{
48+
"type": "aggregate",
49+
"name": "channel_page_hourly_distinct_user_added_deleted",
50+
"groupingColumns": [
51+
{
52+
"type": "long",
53+
"name": "__gran"
54+
},
55+
{
56+
"type": "string",
57+
"name": "channel"
58+
},
59+
{
60+
"type": "string",
61+
"name": "page"
62+
}
63+
],
64+
"virtualColumns": [
65+
{
66+
"type": "expression",
67+
"expression": "timestamp_floor(__time, 'PT1H')",
68+
"name": "__gran",
69+
"outputType": "LONG"
70+
}
71+
],
72+
"aggregators": [
73+
{
74+
"type": "HLLSketchBuild",
75+
"name": "distinct_users",
76+
"fieldName": "user"
77+
},
78+
{
79+
"type": "longSum",
80+
"name": "sum_added",
81+
"fieldName": "added"
82+
},
83+
{
84+
"type": "longSum",
85+
"name": "sum_deleted",
86+
"fieldName": "deleted"
87+
}
88+
]
89+
}
90+
]
91+
```
92+
93+
</details>
94+
95+
A query targeting the aggregated dimensions grouped in the same way uses the projection, for example:
96+
97+
<details>
98+
<summary>Show the query</summary>
99+
100+
```sql
101+
SELECT
102+
TIME_FLOOR(__time, 'PT1H') AS __gran,
103+
channel,
104+
page,
105+
APPROX_COUNT_DISTINCT_DS(user) AS distinct_users,
106+
SUM(added) AS sum_added,
107+
SUM(deleted) AS sum_deleted
108+
FROM your_datasource
109+
GROUP BY
110+
TIME_FLOOR(__time, 'PT1H'),
111+
channel,
112+
page
113+
```
114+
115+
</details>
116+
117+
## Create a projection
118+
119+
You can either create a projection as part of your ingestion or manually add them to an existing datasource.
120+
121+
You can create a projection:
122+
123+
- in the ingestion spec or query for your datasource
124+
- in the catalog for an existing datasource
125+
- in the compaction spec for an existing datasource
126+
127+
In addition to the columns in your datasource, a projection has three components:
128+
129+
- Virtual columns (`spec.projections.virtualColumns`) composed of multiple existing columns from your datasource. A projection can reference an existing column in your datasource or the virtual columns defined in this block.
130+
- Grouping columns (`spec.projections.groupingColumns`) to sort the projection. They must either already exist in your datasource or be defined in `virtualColumns` of your ingestion spec. The order in which you define your grouping columns dictates the sort order for in the projection. Sort order is always ascending.
131+
- Aggregators (`spec.projections.aggregators`) that define the columns you want to create projections for and the aggregator to use for that column. The columns must either already exist in your datasource or be defined in `virtualColumns`.
132+
133+
Note that any projection dimension you create becomes part of your datasource. You need to reingest the data to remove a projection from your datasource. Alternatively, you can use a query context parameter to avoid using projections for a specific query.
134+
135+
### Limitations
136+
137+
When creating a projection, keep the following limitations in mind:
138+
139+
- If your projection includes source columns that are type `float`, you need to use double aggregations, like `doubleSum`, in the projection.
140+
- The aggregator in your projection must match the aggregator in the query for a projection to get used. For example, the output type of the `cardinality` aggregator is different at ingestion time (string) and at query time (long)
141+
- Since the source columns for a projection are unaffected by a projection, storage requirements can increase.
142+
-
143+
144+
### As part of your ingestion
145+
146+
To create a projection at ingestion time, use the [`projectionsSpec` block in your ingestion spec](../ingestion/ingestion-spec.md#projections).
147+
148+
<details>
149+
<summary>Show the ingestion spec</summary>
150+
151+
```json
152+
153+
```
154+
155+
</details>
156+
157+
To create projections for SQL-based ingestion, you need to also have the [`druid-catalog`](../development/extensions-core/catalog.md) extension loaded.
158+
159+
### Manually add a projection
160+
161+
You can define a projection for an existing datasource. We recommend using the [`druid-catalog`](../development/extensions-core/catalog.md) extension, but you can also define the projection in a compaction spec.
162+
163+
The following API call includes a payload with the `properties.projections` block that defines your projections:
164+
165+
<details>
166+
<summary>View the payload</summary>
167+
168+
```json {11,19,39} showLineNumbers
169+
{
170+
"type": "datasource",
171+
"columns": [],
172+
"properties": {
173+
"segmentGranularity": "PT1H",
174+
"projections": [
175+
{
176+
"spec": {
177+
"name": "channel_page_hourly_distinct_user_added_deleted",
178+
"type": "aggregate",
179+
"virtualColumns": [
180+
{
181+
"type": "expression",
182+
"name": "__gran",
183+
"expression": "timestamp_floor(__time, 'PT1H')",
184+
"outputType": "LONG"
185+
}
186+
],
187+
"groupingColumns": [
188+
{
189+
"type": "long",
190+
"name": "__gran",
191+
"multiValueHandling": "SORTED_ARRAY",
192+
"createBitmapIndex": false
193+
},
194+
{
195+
"type": "string",
196+
"name": "channel",
197+
"multiValueHandling": "SORTED_ARRAY",
198+
"createBitmapIndex": true
199+
},
200+
{
201+
"type": "string",
202+
"name": "page",
203+
"multiValueHandling": "SORTED_ARRAY",
204+
"createBitmapIndex": true
205+
}
206+
],
207+
"aggregators": [
208+
{
209+
"type": "HLLSketchBuild",
210+
"name": "distinct_users",
211+
"fieldName": "user",
212+
"lgK": 12,
213+
"tgtHllType": "HLL_4"
214+
},
215+
{
216+
"type": "longSum",
217+
"name": "sum_added",
218+
"fieldName": "added"
219+
},
220+
{
221+
"type": "longSum",
222+
"name": "sum_deleted",
223+
"fieldName": "deleted"
224+
}
225+
]
226+
}
227+
}
228+
]
229+
}
230+
}
231+
```
232+
233+
</details>
234+
235+
In this example, Druid aggregates data into `distinct_user`, `sum_added`, and `sum_deleted` dimensions based on the aggregator that's specified and a source dimension. These aggregations are grouped by the columns you define in `groupingColumns`.
236+
237+
## Use a projection
238+
239+
Druid automatically uses a projection if your query matches a projection you've defined. You can use the following query context parameters to override the default behavior for projections:
240+
241+
- `useProjection`: The name of a projection you defined. The query engine must use this specific projection. If the projection does not match the query, the query fails.
242+
- `forceProjections`: Set to `true` or `false`. Requires the query engine to use a projection. If no projection matches the query, the query fails. Defaults to `false`, which means that Druid uses a projection if there is one that matches your query. If there isn't, Druid processes the query as usual.
243+
- `noProjections`: Set to `true` or `false`. The query engine won't use any projections.
244+
245+
## Compaction
246+
247+
To use compaction on a datasource that includes projections, you need to set the spec type to catalog: `spec.type: catalog`:
248+
249+
<Tabs>
250+
<TabItem value="Coordinator duties">
251+
252+
```json
253+
{
254+
"type": "catalog",
255+
"dataSource": YOUR_DATASOURCE,
256+
"engine": "native",
257+
"skipOffsetFromLatest": "PT0H",
258+
"taskPriority": 25,
259+
"inputSegmentSizeBytes": 100000000000000,
260+
"taskContext": null
261+
}
262+
```
263+
264+
</TabItem>
265+
<TabItem value="Supervisors">
266+
267+
```json
268+
{
269+
"type": "autocompact",
270+
"spec": {
271+
"type": "catalog",
272+
"dataSource": YOUR_DATASOURCE,
273+
"engine": "native",
274+
"skipOffsetFromLatest": "PT0H",
275+
"taskPriority": 25,
276+
"inputSegmentSizeBytes": 100000000000000,
277+
"taskContext": null
278+
},
279+
"suspended": true
280+
}
281+
```
282+
283+
</TabItem>
284+
</Tabs>
285+

‎website/sidebars.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -180,6 +180,7 @@
180180
},
181181
"querying/querying",
182182
"querying/query-processing",
183+
"querying/projections",
183184
"querying/query-execution",
184185
"querying/dart",
185186
"querying/troubleshooting",

0 commit comments

Comments
 (0)