Source

Timeplus integrates with a wide range of systems as data sources, such as Apache Kafka.

You can define one or more sources to set up the background jobs to load data into Timeplus. Please check the Data Ingestion section for more details.

Source API

If you need to call an API to create a source, here are the references.

kafka

refer to https://kafka.apache.org/

Property	Required	Description	Default
brokers	yes	Specifies the list of broker addresses. This is a comma-separated string. such as kafka1:9092,kafka2:9092,kafka3:9092
topic	yes	Specifies the Kafka topic to connect
offset	yes	Specifies the Kafka offset configuration. support latest,earliest
data_type	yes	Specifies the data type to use for creating the stream. support json,text,avro,debezium-json,debezium-json-upsert
group	no	Specifies the Kafka consumer group. use the source uuid with prefix timeplus-source- as the default value if user does not specify it	timeplus-source-<uuid>
sasl	no	Specifies the Simple Authentication and Security Layer (SASL) mechanism for authentication. support none,plain,scram-sha-256,scram-sha-512	none
username	no	Specifies the username for authentication
password	no	Specifies the password for authentication
tls.disable	no	If set to true, disables TLS encryption	false
tls.skip_verify_server	no	If set to true, skips server certificate verification when using TLS	false
schema_registry_address	no	Specifies the URL of the Schema Registry for Kafka, only applies when the data_type is avro
schema_registry_api_key	no	Specifies the API key for Schema Registry authentication
schema_registry_api_secret	no	Specifies the API secret for Schema Registry authentication

stream_generator

a source that generates random data for test

Property	Required	Description	Default
template	yes	Specifies the template used to generate data, support iot,user_logins,devops
interval	no	Specifies the event interval. for example 200ms	200ms

websocket

refer to https://developer.mozilla.org/en-US/docs/Web/API/WebSocket

Property	Required	Description	Default
url	yes	Specifies the URL of the websocket server url
open_message	no	An optional message to send to the server upon connection.
open_message_type	no	An optional flag to indicate the data type of open_message. support text binary	text
data_type	yes	Specifies the data type to use for creating the stream. support json,text,

nats

refer to https://docs.nats.io/nats-concepts/what-is-nats

Property	Required	Description	Default
url	yes	A list of URLs to connect to. for example [nats://127.0.0.1:4222]
subject	yes	A subject to consume from. Supports wildcards for consuming multiple subjects.
queue	no	An optional queue group to consume as.
nak_delay	no	An optional delay duration on redelivering a message when negatively acknowledged.
prefetch_count	no	The maximum number of messages to pull at a time.	524288
data_type	yes	Specifies the data type to use for creating the stream. support json,text,
tls.disable	no	If set to true, disables TLS encryption	false
tls.skip_verify_server	no	If set to true, skips server certificate verification when using TLS	false

nats_jetstream

refer to https://docs.nats.io/nats-concepts/jetstream

Property	Required	Description	Default
url	yes	A list of URLs to connect to. for example [nats://127.0.0.1:4222]
subject	yes	A subject to consume from. Supports wildcards for consuming multiple subjects.
queue	no	An optional queue group to consume as.
durable	no	Preserve the state of your consumer under a durable name.
stream	no	A stream to consume from. Either a subject or stream must be specified..
deliver	no	Determines which messages to deliver when consuming without a durable subscriber. support all last	all
bind	no	A bool indicates that the subscription should use an existing consumer.
ack_wait	no	The maximum amount of time NATS server should wait for an ack from consumer.	30s
max_ack_pending	no	The maximum number of outstanding acks to be allowed before consuming is halted.	1024
data_type	yes	Specifies the data type to use for creating the stream. support json,text,
tls.disable	no	If set to true, disables TLS encryption	false
tls.skip_verify_server	no	If set to true, skips server certificate verification when using TLS	false

pulsar

refer to https://pulsar.apache.org/

Property	Required	Description	Default
topic	yes	Specifies the topic of the pulsar to connect to
broker_url	yes	Specifies the URL of the broker to connect to
auth_type	yes	Specifies the authentication type to use. support ,oauth2,token
auth_params	no	Specifies authentication parameters as key-value pairs	{}

livepeer

refer to https://livepeer.org/

Property	Required	Description	Default
interval	yes	Specifies the pulling interval to livepeer api. for example 300s
api_key	yes	Specifies the API key of livepeer
data_type	yes	Specifies the data type to use for creating the stream. support json,text

ably

refer to https://ably.com/

Property	Required	Description	Default
api_key	yes	The api_key property is a string used to authenticate and authorize access to the Ably service. It represents the API key associated with the Ably account, which is required for making authenticated requests to Ably's services
channel	yes	The channel property specifies the name of the channel to which messages will be sent or from which messages will be received. Channels in Ably are used to group and categorize messages. This property allows you to select the target channel for your interactions