SymetryML Documentation
Streams

Poll stream

Fetch the next available batch of records from a Kafka topic as a one-off `DataFrame`, independent of the create/start/stop/delete stream lifecycle above (NATS is not supported here). Each poller is cached server-side for 2 minutes of inactivity, keyed by the DSInfo's `name`, so repeated polls against the same name continue from where the previous poll left off rather than re-reading from the start. Two distinct call patterns exist, and they behave very differently — pick based on which matches your situation, not by symmetry with the request body's shape: * **`sid` query param, no body**: look up an already-registered Data Source (e.g. one created via [Create stream](/docs/api-reference/streams/projects-project-streams-create-post)). This is the clean path: on success, returns a plain `200` with the batch. Requires the Data Source to already exist under that name — this call does **not** register anything itself. * **request body, no `sid`**: the real usage pattern from the CLI's stream-poll commands, which always send DSInfo `type: "kafka-poll"`. **Only `kafka`/`nats` are accepted as valid stream types — `kafka-poll` never passes** — so this call unconditionally returns `statusCode: 400`, `statusString: "Preview invalid DS Stream type [kafka-poll]"`. However, the server does not return early on that failed validation — it falls through and polls Kafka anyway, so **`values.dataframe` still carries a real batch of records** on the very same 400 response. Treat `values.dataframe`'s presence, not `statusCode`, as the actual success signal for this call pattern. This body-based call also does not register the DSInfo as a Data Source (the registration branch is skipped along with the rest of the failed-validation path), so a later `sid`-only poll under the same name will not find it. If both `sid` and a body are supplied, `sid` wins and the body is ignored entirely — its (in)validity has no effect on that request.

POST
/{user}/streams/poll

Fetch the next available batch of records from a Kafka topic as a one-off DataFrame, independent of the create/start/stop/delete stream lifecycle above (NATS is not supported here). Each poller is cached server-side for 2 minutes of inactivity, keyed by the DSInfo's name, so repeated polls against the same name continue from where the previous poll left off rather than re-reading from the start.

Two distinct call patterns exist, and they behave very differently — pick based on which matches your situation, not by symmetry with the request body's shape:

  • sid query param, no body: look up an already-registered Data Source (e.g. one created via Create stream). This is the clean path: on success, returns a plain 200 with the batch. Requires the Data Source to already exist under that name — this call does not register anything itself.
  • request body, no sid: the real usage pattern from the CLI's stream-poll commands, which always send DSInfo type: "kafka-poll". Only kafka/nats are accepted as valid stream types — kafka-poll never passes — so this call unconditionally returns statusCode: 400, statusString: "Preview invalid DS Stream type [kafka-poll]". However, the server does not return early on that failed validation — it falls through and polls Kafka anyway, so values.dataframe still carries a real batch of records on the very same 400 response. Treat values.dataframe's presence, not statusCode, as the actual success signal for this call pattern. This body-based call also does not register the DSInfo as a Data Source (the registration branch is skipped along with the rest of the failed-validation path), so a later sid-only poll under the same name will not find it.

If both sid and a body are supplied, sid wins and the body is ignored entirely — its (in)validity has no effect on that request.

Authorization

SymetryMLAuth
AuthorizationBearer <token>

HMAC-SHA256 signature-based authentication: requests are signed with your secret key and sent with the Customer-ID, Sym-date, Authorization, Content-MD5 and sym-version headers. See SymetryML REST API Security for the signature algorithm and a complete example.

In: header

Path Parameters

user*string

User/Customer identifier

Query Parameters

sid?string

Name of an existing kafka Data Source to poll. See description for interaction with the request body.

fromBeginning?boolean

Only applies the first time a given sid/DSInfo is polled (i.e. when its poller is first created).

Defaultfalse

Request Body

application/json

TypeScript Definitions

Use the request body type in TypeScript.

body?string

Encrypted DSInfo

Response Body

application/json

application/json

application/json

curl -X POST "https://example.com/c1/streams/poll" \  -H "Content-Type: application/json" \  -d '1TYsk+/zj32PJvmeT7FY7mT9yn0IpjSN2KcM1mRkpZEwbDiTsm04CcXJYQWm2r42q0JZnvZ43pFg9hutZhonwtVZkg5x5W4Bk2rhZoAGhD1iCSREqWLRKgPvv2Gdw3yHJj0wEx9O70d21p9VF/uEDXtMsIgFSdZeuLNStZJoUYvWuSmDxMRGznKILNRw8FNTIdNlQ61TWBrTBcWVKFqQoA=='

Polling a sid registered moments earlier via Create stream. A real response can hold hundreds of rows; abridged to 2 here.

{  "statusCode": 200,  "statusString": "OK",  "values": {    "dataframe": {      "attributeNames": [        "attr0",        "attr1",        "attr2",        "attr3",        "attr4",        "attr5",        "attr6",        "attr7"      ],      "data": [        [          "0.7000000000000001",          "0.4",          "0.2",          "0.30000000000000004",          "1.0",          "0.0",          "1.0",          "0.9"        ],        [          "0.9",          "0.9",          "0.4",          "0.30000000000000004",          "0.30000000000000004",          "0.5",          "0.1",          "0.8"        ]      ],      "attributeTypes": [        "C",        "C",        "C",        "C",        "C",        "C",        "C",        "C"      ],      "errorHandling": 1    }  }}

The server sets this 400 on the failed type-validation check, then falls through and polls Kafka anyway rather than returning early — so a real batch (hundreds of rows in a typical response, abridged here) rides along with the error status.

{  "statusCode": 400,  "statusString": "Preview invalid DS Stream type [kafka-poll]",  "values": {    "dataframe": {      "attributeNames": [        "attr0",        "attr1",        "attr2",        "attr3",        "attr4",        "attr5",        "attr6",        "attr7"      ],      "data": [        [          "0.7000000000000001",          "0.4",          "0.2",          "0.30000000000000004",          "1.0",          "0.0",          "1.0",          "0.9"        ],        [          "0.9",          "0.9",          "0.4",          "0.30000000000000004",          "0.30000000000000004",          "0.5",          "0.1",          "0.8"        ]      ],      "attributeTypes": [        "C",        "C",        "C",        "C",        "C",        "C",        "C",        "C"      ],      "errorHandling": 1    }  }}

{  "statusCode": 500,  "statusString": "Generic Exception [org.apache.kafka.common.KafkaException: Failed to construct kafka consumer]",  "values": {}}

Stop and evict a polled stream DELETE

Stop and remove a [Poll stream](/docs/api-reference/streams/streams-poll-post) poller for the given `sid` from the server-side cache. This does not touch the underlying `kafka` Data Source's lifecycle managed by [Create stream](/docs/api-reference/streams/projects-project-streams-create-post)/ [Start stream](/docs/api-reference/streams/projects-project-streams-dsname-start-get)/ [Stop stream](/docs/api-reference/streams/projects-project-streams-dsname-stop-get)/ [Delete stream](/docs/api-reference/streams/projects-project-streams-dsname-delete) — it only tears down the on-demand poller and, if the `sid` also identifies a registered Data Source, deletes that Data Source too.

Preview stream POST

Fetch a one-off sample `DataFrame` from a `kafka`/`nats` source without creating or registering any Data Source — purely a preview, with no side effects and no interaction with the create/start/stop/delete stream lifecycle or with [Poll stream](/docs/api-reference/streams/streams-poll-post)'s cache.