Advanced Data Pipeline

The Advanced Data Pipeline (AKA Advanced Mode) is an advanced version of our Data Explorer. It is essentially a visual workflow for processing data to build a chart. Merge and transform any number of datasets at any point in the Pipeline using the simple drag-and-drop interface. You’ll find Advanced Mode offers more flexibility when transforming your data, as there are some actions that can’t be done in Basic Mode.

Some common use cases for Advanced Mode include:

  • Reordering your Datasets
  • Performing multiple merges
  • Testing and troubleshooting individual Pipeline Steps
  • Undoing and redoing changes to your Pipeline

The example below shows how a user can merge two datasets, apply a transformation, then merge a third dataset. Merge two datasets, apply a transformation, then merge a third dataset.

Toggle to Advanced Mode

When creating a new chart from either an existing dashboard or via the Explore tab in the top navigation bar, click Advanced above the Dataset Interface to switch to the Advanced Data Pipeline view.

Toggle to Advanced Pipeline

Advanced Data Pipeline Overview Diagram

Once in Advanced Mode, you’ll see an empty Database Query node, which you can edit (similar to your Dataset in Basic Mode), and a Chart node on your canvas.

Note: If your Advanced Data Pipeline isn’t compatible with Basic Mode, the Basic toggle button will be disabled.

New canvas in Advanced Mode has empty Database Query node and a Chart node

Build your Data Pipeline

Charts are built by first dragging one or more Database Query nodes onto the canvas, combining those nodes using one or more Merge Step nodes, and connecting the final node to the Chart node. Add Transformation Step nodes as needed to modify your dataset results. Just like in Basic Mode, you start by editing your initial Database Query node. From there, you can add more nodes to your canvas and reorder them to get your desired result set.

Add a node

To add a node to the canvas, click and drag it from the left sidebar onto the canvas.

Connect nodes

When adding new nodes, connections to the new node are already made if you placed it on the connection of two existing nodes. You’ll know this will happen when you drag the new node over the connection and the connection line turns from solid to dashed.

Use the auto-connect feature when adding new nodes

But maybe you want to manually connect nodes together–you can do that too. Let’s say you want the result of Node A to pass to Node B. To connect them, you’d do the following:

  1. Click the output square of Node A then hover anywhere on Node B.
  2. Node B will be highlighted in blue to show the connection is recognized. Nodes to which you can’t connect will be greyed out.
  3. Finally, click on Node B to complete the connection.

Manually connect nodes in Advanced Mode

If a node’s output is already connected to another node, its connector square won’t be visible. You can only connect a node’s output to one other node at a time. However, a Merge Step node can accept multiple input connections.

Disconnect nodes

To disconnect two nodes, hover over the line connecting the nodes and click the Scissors that appear.

Click scissors to disconnect nodes

Preview Data drawer

You can preview a specific node’s data by expanding the Preview Data drawer. To open the Preview Data drawer, click Preview Data at the bottom-right corner of the canvas. Whenever you click a node in your Pipeline, Preview Data will show the selected node’s data.

  • For Database Query nodes, you can view the Output Data and the executed SQL query.
  • For Merge Step nodes, you can only view the Output Data.
  • For Transformation Step nodes, you can view the Input and Output data.
  • For the Chart node, you can view the Chart Data, which is the final result set of your Pipeline.

Preview Data

Run Selected Node

Whenenver you’re editing a node, you can click Run Selected Node, which appears next to the Run Pipeline button, to run the Pipeline up to that point and view its updated result set in the Preview Data drawer. This can be especially useful for testing and troubleshooting a part of your Pipeline separately.

Run selected Node

Execute your Data Pipeline

To run your queries and update your entire Pipeline, click Run Pipeline at the bottom-right corner of the Data Explorer. Your Chart node will update with the new result set.

Run Advanced Pipeline Example