Pipeline File

Optimus Mine pipelines are described using a YAML file. Each pipeline is made up of one or more tasks, with a task being one of Optimus Mine’s engines.


  - name: Custom Input
    engine: input
    text: '{"foo":"bar"}'
    decoder: json
      type: object
          type: string
  - name: Log data
    engine: debug



Optionally give your task a name for use in logs.



The name of the Optimus Mine engine. Other fields depend on what engine you are using. See the links below for more information.



Engine Description
Debug Write data to logs for debugging pipelines.
HTTP Make an HTTP request.
Input Custom user input.
JavaScript Create a custom parser using JavaScript.
Parse Parse arbitrary data.
Template Render text using liquid templates.


Engine Description
Amazon Athena: Query Query data in Amazon S3 using Amazon Athena.
Google BigQuery: Export Export data to Google BigQuery.
Google BigQuery: Query Query a Google BigQuery table.
Yandex ClickHouse: Export Export data to Yandex ClickHouse.
Yandex ClickHouse: Query Query a Yandex ClickHouse table.
Azure Cosmos: Query Query an Azure Cosmos database using SQL.
Amazon DynamoDB: Query Query a DynamoDB database.
Amazon DynamoDB: Export Export data to a DynamoDB database.
Elastic Search: Export Export data to Elasticsearch.
Firebird SQL: Export Export data to a Firebird SQL database.
Firebird SQL: Query Query a Firebird SQL database.
Google Firestore: Query Query a Firestore database.
Google Firestore: Export Export data to a Firestore database.
SAP HANA: Export Export data to an SAP HANA database.
SAP HANA: Query Query an SAP HANA database.
Alibaba Cloud MaxCompute: Export Export data to Alibaba Cloud MaxCompute.
Alibaba Cloud MaxCompute: Query Query an Alibaba Cloud MaxCompute database.
MongoDB: Query Query a MongoDB database.
MongoDB: Export Export data to a MongoDB database.
MySQL: Export Export data to a MySQL database.
MySQL: Query Query a MySQL database.
Oracle: Export Export data to an Oracle database.
Oracle: Query Query an Oracle database.
PostgreSQL: Export Export data to a PostgreSQL database.
PostgreSQL: Query Query a PostgreSQL database.
Snowflake: Export Export data to a Snowflake database.
Snowflake: Query Query a Snowflake database.
Microsoft SQL Server: Export Export data to a Microsoft SQL Server database.
Microsoft SQL Server: Query Query a Microsoft SQL Server database.
SAP ASE: Export Export data to an SAP ASE database.
SAP ASE: Query Query an SAP ASE database.


Engine Description
Azure Service Bus: Subscribe Import data from Azure Service Bus.
Google Cloud Pub/Sub: Subscribe Import data from Google Cloud Pub/Sub.
Kafka: Subscribe Import data from Kafka.
NATS: Subscribe Import data from NATS.
RabiitMQ: Subscribe Import data from RabbitMQ.
SQS: Subscribe Import data from AWS SQS.


Engine Description
AWeber: Subscribers Extract subscribers data from AWeber.
Constant Contact: Contacts Extract Contacts data from Constant Contact.
LinkedIn: Ad Analytics LinkedIn: Ad Analytics
Mailchimp: Members Extract members data from Mailchimp.
Pipedrive: Deals Extract deals data from Pipedrive.
Pipedrive: Leads Extract leads data from Pipedrive.
Pipedrive: Organizations Extract organizations data from Pipedrive.
Pipedrive: Persons Extract persons data from Pipedrive.


Engine Description
Azure Blob Storage: Read Import data from Azure Blob Storage.
Azure Blob Storage: Write Upload data to Azure Blob Storage.
Dropbox: Read Upload data to Dropbox.
Dropbox: Write Upload data to Dropbox.
Local File: Read Import data from a local file.
Local File: Write Save data to a local file.
Google Cloud Storage: Read Import data from Google Cloud Storage.
Google Cloud Storage: Write Upload data to Google Cloud Storage.
S3: Read Import data from S3-compatible storage services.
S3: Write Upload data to S3-compatible storage services.
SFTP: Read Import data from an FTP server.
SFTP: Write Upload data to an FTP server.