Pipeline File

Example
Properties
- name
- engine
  - Core
  - Database
  - Pubsub
  - Marketing
  - Storage

Optimus Mine pipelines are described using a YAML file. Each pipeline is made up of one or more tasks, with a task being one of Optimus Mine’s engines.

Example

tasks:
  - name: Custom Input
    engine: input
    text: '{"foo":"bar"}'
    decoder: json
    schema:
      type: object
      properties:
        foo:
          type: string
  - name: Log data
    engine: debug

Properties

name

Optionally give your task a name for use in logs.

Required: No

engine

The name of the Optimus Mine engine. Other fields depend on what engine you are using. See the links below for more information.

Required: Yes

Core

Engine	Description
Debug	Write data to logs for debugging pipelines.
HTTP	Make an HTTP request.
Input	Custom user input.
JavaScript	Create a custom parser using JavaScript.
Parse	Parse arbitrary data.
Template	Render text using liquid templates.

Database

Engine	Description
Amazon Athena: Query	Query data in Amazon S3 using Amazon Athena.
Google BigQuery: Export	Export data to Google BigQuery.
Google BigQuery: Query	Query a Google BigQuery table.
Yandex ClickHouse: Export	Export data to Yandex ClickHouse.
Yandex ClickHouse: Query	Query a Yandex ClickHouse table.
Azure Cosmos: Query	Query an Azure Cosmos database using SQL.
Amazon DynamoDB: Query	Query a DynamoDB database.
Amazon DynamoDB: Export	Export data to a DynamoDB database.
Elastic Search: Export	Export data to Elasticsearch.
Firebird SQL: Export	Export data to a Firebird SQL database.
Firebird SQL: Query	Query a Firebird SQL database.
Google Firestore: Query	Query a Firestore database.
Google Firestore: Export	Export data to a Firestore database.
SAP HANA: Export	Export data to an SAP HANA database.
SAP HANA: Query	Query an SAP HANA database.
Alibaba Cloud MaxCompute: Export	Export data to Alibaba Cloud MaxCompute.
Alibaba Cloud MaxCompute: Query	Query an Alibaba Cloud MaxCompute database.
MongoDB: Query	Query a MongoDB database.
MongoDB: Export	Export data to a MongoDB database.
MySQL: Export	Export data to a MySQL database.
MySQL: Query	Query a MySQL database.
Oracle: Export	Export data to an Oracle database.
Oracle: Query	Query an Oracle database.
PostgreSQL: Export	Export data to a PostgreSQL database.
PostgreSQL: Query	Query a PostgreSQL database.
Snowflake: Export	Export data to a Snowflake database.
Snowflake: Query	Query a Snowflake database.
Microsoft SQL Server: Export	Export data to a Microsoft SQL Server database.
Microsoft SQL Server: Query	Query a Microsoft SQL Server database.
SAP ASE: Export	Export data to an SAP ASE database.
SAP ASE: Query	Query an SAP ASE database.

Pubsub

Engine	Description
Azure Service Bus: Subscribe	Import data from Azure Service Bus.
Google Cloud Pub/Sub: Subscribe	Import data from Google Cloud Pub/Sub.
Kafka: Subscribe	Import data from Kafka.
NATS: Subscribe	Import data from NATS.
RabiitMQ: Subscribe	Import data from RabbitMQ.
SQS: Subscribe	Import data from AWS SQS.

Marketing

Engine	Description
AWeber: Subscribers	Extract subscribers data from AWeber.
Constant Contact: Contacts	Extract Contacts data from Constant Contact.
LinkedIn: Ad Analytics	LinkedIn: Ad Analytics
Mailchimp: Members	Extract members data from Mailchimp.
Pipedrive: Deals	Extract deals data from Pipedrive.
Pipedrive: Leads	Extract leads data from Pipedrive.
Pipedrive: Organizations	Extract organizations data from Pipedrive.
Pipedrive: Persons	Extract persons data from Pipedrive.

Storage

Engine	Description
Azure Blob Storage: Read	Import data from Azure Blob Storage.
Azure Blob Storage: Write	Upload data to Azure Blob Storage.
Dropbox: Read	Upload data to Dropbox.
Dropbox: Write	Upload data to Dropbox.
Local File: Read	Import data from a local file.
Local File: Write	Save data to a local file.
Google Cloud Storage: Read	Import data from Google Cloud Storage.
Google Cloud Storage: Write	Upload data to Google Cloud Storage.
S3: Read	Import data from S3-compatible storage services.
S3: Write	Upload data to S3-compatible storage services.
SFTP: Read	Import data from an FTP server.
SFTP: Write	Upload data to an FTP server.