Early Preview of Pipedream, a tool to build data pipelines using Node

Hi y’all,My name is Dylan and I’m a founder at Pipedream. We’re building a tool to help you collect, transform and route data using Node.js, for free, without managing any data infrastructure or cloud functions.I wanted to share an early preview of the app to get feedback from the JS community. We’re humbled to release this publicly, and open to any and all feedback for how we can improve it to help you build cool stuff.I recorded a short video showing you how to create your first pipeline on Pipedream. I also wanted to share a few template pipelines — you can fork any of these into your own account to run them — as examples of things you can do today:Send new Github issues to Slack (pipeline from video tutorial)Send any webhook to Discord (Stripe example)Query Segment data using SQL in < 5 minutesSend webhook / HTTP data to S3Send JSON to a pipeline and query it with SQL in minutes, without setting up a database or other data warehouse infrastructure.We’ve adopted a notebook-like interface for editing code, similar to tools like Observable or Runkit. All notebooks (pipelines) are public, so anyone can find and use a notebook you’ve written. The data you send to your pipelines is completely private.Think of us as a Github for data pipelines: find a notebook that solves your use case, fork it, and run it in your own account.A few more things to know:Running pipelines is free. In turn, we’ve imposed some limits on pipeline volume and the runtime environment for Node code during the preview. If you find these limits restrictive, let us know!All pipeline code is public, but the data you send to pipelines is private. Since code is public, you should store any sensitive values like API keys in environment variables.We support Node 10. You can use npm packages just by require-ing them in a code cell. See the docs on code cells for more detail on how to use Node code on Pipedream.We have a long way to go and have a lot of features on the way, but in the meantime, give it a try, build some pipelines, and let us know what you think!

Submitted May 31, 2019 at 10:52PM by dylburger-pipedream