Flow Table (Parquet)
| Purpose | Explore this guide to discover what is the Flow Table (Parquet) and how to create one. |
|---|
| Last Updated | August 20, 2024 |
|---|
What is Flow Table (Parquet)?
The Flow Table (Parquet) Table type serves as a transformation process, transforming any information going through the flow into a parquet file.
Parquet serves as the backend file format for Query, but it remains hidden from the end user's view.
For more information in Parquet files follow link below:
Setting up a Flow Table (Parquet)
When you're setting up a Flow Table (Parquet), you'll be prompted to provide several key pieces of information:
-
Topic: This is where you specify the source from which the data is being extracted.
-
Transform: This allows you to choose a Transform Brick to modify data and meet specific requirements.
-
Refresh Interval: Here, you define how often the data in your Flow Table (Parquet) should be updated.
-
Sample Size: Specify the size of the sample data that will be used in your Flow Table (Parquet).
-
Batch Size: This is where you determine the size of data batches to be processed.
-
Retention Policy: Define how long the data should be retained in your Flow Table (Parquet).
-
Schema: Describe the structure and organization of the data.
-
Pick a Schema: This function is used to establish a mapping for the schema.
The next step after providing all the necessary information is to proceed to press the "CREATE TABLE" button, and there you have it! you have created your first Flow Table (Parquet).
@rowid
Flow Tables (Parquet) Tables automatically create a column called @rowid, These are unique identifiers given by the system to events within the topic under consideration and their @rowid numbers correspond with the sequence of the occurrence of the event. The first event is given @rowid 0, The second event is given @rowid 1 and so on and so forth for the next events. This numbering makes events correspond with their occurrence in the topic and hence are arranged in order.