No Data Left Behind: Introducing Singer, Open Source ETL
Posted by on March 1, 2017 Data, Data Stack, Database, Data Governance
[Editor’s Note: Our partner Stitch is introducing Singer, an open source project for simple, composable ETL. Singer enables any data source to be analyzed in Chartio - regardless of whether or not you’re a Stitch customer. Read on for full details.]
For our customers to gain insight into their organization, data needs to be consolidated from different tools and sources into a data warehouse. With that, many of our customers use our partner Stitch to handle data consolidation for their organization. While Stitch integrates with more than 50 different data sources, we know that there are many more places where your data might live.
When our customers need a data source that isn’t supported by commercial ETL (which stands for extract, transform, load) tools, their only option has been to write a custom script to pull the data themselves. While writing a script for a one-time data pull isn’t too hard, we’ve found that over the long term it creates a big maintenance burden. APIs change, a different data warehouse is chosen, servers go down, cron jobs stop running, API rate limits get hit, and so on. This is why we recommend commercial ETL tools in the first place, but the problem isn’t totally solved if only five of your six data sources are supported.
Stitch is introducing Singer to solve that problem. Singer is a better way to write and collaborate on ETL scripts, and it includes three parts:
- Taps, which pull data from sources
- Targets, which send data to destinations
- A JSON-based format for communication between taps and targets
All taps and targets can be mixed and matched, so changing the destination you’re loading data into is easy. Since it’s all open source, the community can leverage each other’s improvements.
Singer integrations can be run on hardware that you manage, but Stitch can run any integration written in the Singer format. When you use Stitch for your Singer integrations, you’ll get auto-scaling, a secure infrastructure, credential management, monitoring, and alerting.
Check out Singer.io and analyze all of your data in Chartio.
