What are Tables and their types

Purpose	In this document, we will delve deeper into a significant feature offered by the Raven Portal, aiming to enhance user efficiency and adaptability. This feature, referred to as Tables, assists in organizing incoming data effectively.

Last Updated	July 29, 2024

---

Tables: Structuring and Managing Data

Tables play a pivotal role in organizing and managing incoming data sources within the Raven Portal. They enable you to structure the data in a tabular format, facilitating easy manipulation and retrieval.

Supported Data Sources

Raven allows the creation of tables using various data sources:

1. Data straight from the flows

2. CSV files

3. JSON files

4. External data sources using dynamic table bricks, for example:

   • API

   • Elasticsearch

   • SQL databases

To access your Tables, navigate to the "Settings" section and select "Tables." Here, you can find essential information about your existing Tables:

• Name
• Description
• Type
• Record Count
• Data Size
• Creation Date

To create a new Table, click the "Create Table" button in the upper right corner. This action opens a configuration pane where you can provide details:

• Alias: A reference name for the table
• Description (optional)
• Table Type: Various types available, discussed later

Additional configuration options may vary depending on the chosen table type.

Save and create the Table by clicking the "Create Table" button at the bottom of the configuration pane.

Tables can also be created programmatically through Python and integrated into Raven Flows.

Transformations: Enhancing Data Tables

Transformations enable you to modify data to meet specific requirements. Depending on the transformation type, you can select the Transformation Brick from the Table types and provide details such as topic, refresh interval, sample size, batch size, retention policy, and schema.

To explore the capabilities of a particular transformation brick, navigate to the code section of Brick Manager V2.

Creating and Managing Transformations

To create and manage Transformations:

1. Go to "Settings" and navigate to "Brick Management v2."

2. Click the "Create Brick" button to configure a new brick. Provide a name, description, type, and optionally add Python code by selecting the "code" pane and clicking "Add new file."

3. Save your changes by clicking the "Save" button in the top right corner.

Timestamp parameter input

You can add a timestamp parameter to the different types of tables by including the necessary code in the schema. Here’s an example:

{
    "fields":[
        {
            "name": "timestampexample",
            "nullable": true,
            "dict_id": 0,
            "dict_is_ordered": false,
            "metadata": {},
            "data_type": {
                    "Timestamp": [
                        "Nanosecond",
                        "UTC"
                    ]
            }
        }
    ]
}

Types of Tables

Raven offers different types of Tables to suit your data needs:

1. Flow Table (Parquet)

2. Mesh Table

3. Flow Table

4. File Table

5. Mapping Table