Skip to content

Data Sources

Data sources are a catalog of data endpoints to be used as sources and sinks. Data providers allows you to register the provider using your security credentials, then use that provider for a source or sink in SQLStreamBuilder.

Internal Kafka Source

You can always use the internal Kafka data source for development, experimentation, and when you don't already have a Kafka cluster at your disposal. You do not need to pre-configure this source, it's built into every SQLStreamBuilder cluster by default. It has the following characteristics:

  • Built in Apache Kafka cluster
  • Can produce to it via REST API
  • Can create topics, list topics, and see basic configuration via REST API
  • Can consume from it using Eventador only. The Kafka endpoints are not exposed otherwise

You must first create a Cluster in order to produce or consume from the Internal Kafka Source. See the API docs and this Python example for producing data to this source.

To consume from this source, create a new Virtual Table Source and select the cluster and topic you are producing data to. The Kafka Cluster will be named the same name as the Cluster you created.

AWS MSK (Managed Streaming Kafka)

To register AWS MSK as a data provider:

It's important to note that because AWS MSK doesn't have any public endpoints, it must be VPC peered to your Eventador VPC for connectivity to work. If you need help with this, don't hesitate to contact support and we will lead you through getting the VPC's peered.

  • Click on the Data Sources menu option.
  • Click on Add Kafka Provider.
  • Provide a name for the provider. This can be any logical name that makes sense for your use case(s) and must be unique.
  • Enter the broker:port connection string for your cluster. You can find this information in the AWS MSK console, by clicking on Client Access. Check out the AWS documentation for specifics.
  • Click the + button to add the connect string, and then the Save Changes button.
  • You should see your new provider in the list of registered providers. You can now see the provider name in the list for sources and sinks.

Confluent Cloud

To register Confluent Cloud (CC) as a data provider:

  • Ensure you have your CC SASL/SSL username and password available.
  • Ensure you have your CC bootstrap.servers value available.
  • Click on the Data Sources menu option.
  • Click on Add Kafka Provider.
  • Provide a name for the provider. This can be any logical name that makes sense for your use case(s) and must be unique.
  • Enter the bootstrap.servers value from CC into the Brokers text field
  • Select SASL/SSL for the connection protocol.
  • Select plain for SASL Mechanism.
  • Enter the CC username and password.
  • Click Save changes. You should see the new provider in the list of registered providers.

source{: style="width: 350px;"}

Database Sources

Various popular databases can be used as a source for data on the Eventador Platform. Debezium provides a stream of changes from various popular databases. Once streamed to Kafka they can be easily consumed like any other source in SQLStreamBuilder. Supported databases (not exhaustive) are:

  • PostgreSQL
  • MongoDB
  • Oracle
  • MySQL
  • Cassandra
  • DB2
  • SQL Server

You can also see this simple tutorial featuring streaming changes from MongoDB directly to SQLStreamBuilder.