# Data Source API (/docs/symetryml-rest-client/rest-documentation/data-source-api)
## Using a Data Source [#using-a-data-source]
In SymetryML, a data source is an abstraction of a CSV file that resides somewhere and that can be used by:
* SymetryML projects to learn new data
* Models to make predictions and assessments
* Encoder to update their internal encoding table
SymetryML supports various types of data sources:
* Secure File Transfer Protocol (SFTP)
* HTTP/HTTPS URL
* Amazon Simple Storage Service (S3)
* Microsoft Azure Blob Storage
* Google Cloud Storage
* Oracle OCI Object Storage
* Amazon RedShift
* Snowflake
* Spark Processing: Amazon S3, Google cloud storage, Oracle OCI object storage and Microsoft Azure blob storage data sources can be processed in parallel leveraging a Spark Cluster.
* SymetryML data source plugins
* JDBC
* Local Data Source, that is allows to browse the local file system of the jetty web server with same privileges as the user running the Jetty web server.
**Streaming Data Sources:** For real-time streaming data sources such as Kafka and NATS, please refer to the [Stream Data Source API](/docs/symetryml-rest-client/rest-documentation/stream-data-source-api) documentation.
To use a data source, create a JSON data structure described in [DSInfo](/docs/symetryml-rest-client/rest-documentation/appendix-a-json-data-structure-schema#dsinfo-json) that contains the fields in table [below](/docs/symetryml-rest-client/rest-documentation/data-source-api#required-dsinfo-fields).
### Required DSInfo Fields [#required-dsinfo-fields]
| Field | Description |
| -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **type** | Type of Data Source
- Secure FTP (SFTP) data source = sftp
- HTTP/HTTPS data source = http
- Amazon S3 = s3
- Oracle OCI Object Storage with S3 Compatibility = s3oci
- Google Clound Storage = gcs
- Amazon Redshift = redshift
- Spark Data Source = please see the Spark Data Source type section for the matrix of all possible data source names involving Spark Processing.
- Data Source Plug ins. - jdbc
- Snowflake = snow
- Local file = localfile
- Amazon Elastic Map Reduce = emr
- Microsoft Azure Blob Storage = abs
- Streaming Data Sources (Kafka, NATS) = kafka or nats - see Stream Data Source API
|
| **name** | Name of the data source. |
| **info** | Hash Map Containing Additional Information Based on Data Source. Please consult the [next section](/docs/symetryml-rest-client/rest-documentation/data-source-api#additional-information-stored-in-data-source) for details about this field. |
### Additional Information Stored in Data Source [#additional-information-stored-in-data-source]
The `info` field of a data source contains specific information based on the type of data source. The following tables describes what they are for each different type of data source
#### HTTP/HTTPS Data Source [#httphttps-data-source]
| field | Description |
| ------ | -------------------------- |
| `path` | http\:// or https\:// URL. |
#### Secure FTP (SFTP) [#secure-ftp-sftp]
| field | Description |
| ------------ | ------------------------------------------------- |
| `path` | path to the file on the server. |
| `sftpuser` | user name used to connect to the SFTP server. |
| `sftppasswd` | user password used to connect to the SFTP server. |
| `sftphost` | host to which you want to connect. |
#### Amazon S3 [#amazon-s3]
| field | Description |
| ------------- | ---------------------------------------------------------- |
| `path` | path to file on the server, excluding the Amazon S3 bucket |
| `s3accessKey` | Amazon S3 access key to use to connect to S3. |
| `s3secretKey` | Amazon S3 secret key to use to connect to S3. |
| `s3bucket` | Amazon S3 bucket to use. |
#### Oracle OCI Object Storage with S3 Compatibility [#oracle-oci-object-storage-with-s3-compatibility]
Please consult [https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) for details on how to configure your Oracle OCI account so that the Amazon S3 Compatibility API can be used.
| field | Description |
| -------------- | ------------------------------------------------------------------------------------------- |
| `path` | path to file on the server, excluding the Oracle OCI Object Storage bucket |
| `s3accessKey` | Oracle Oracle OCI Object Storage access key to use to connect to Oracle OCI Object Storage. |
| `s3secretKey` | Oracle Oracle OCI Object Storage secret key to use to connect to Oracle OCI Object Storage. |
| `s3bucket` | Oracle Oracle OCI Object Storage bucket to use. |
| `ocinamespace` | Oracle Oracle OCI Object Storage namespace |
| `ociregion` | Oracle Oracle OCI Object Storage region |
#### Google Cloud Storage [#google-cloud-storage]
| field | Description |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `path` | Path to file |
| `gcsaccessKey` | GCS [HMAC](https://cloud.google.com/storage/docs/authentication/hmackeys) Access Key |
| `gcssecretKey` | GCS [HMAC](https://cloud.google.com/storage/docs/authentication/hmackeys) Secret Key |
| `gcsbucket` | GCS Bucket |
| `gcsproject` | GCS Project |
| `gcsmarker` | Optional marker parameter indicating where in the GCS bucket to begin listing. The list will only include keys that occur lexicographically after the marker. |
| `gcsdelimiter` | GCS File/Folder delimiter. */* is used by default |
#### Microsoft Azure Blob Storage [#microsoft-azure-blob-storage]
| field | Description |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `path` | Path to file |
| `azure.credentials.connection.string` | Connection string that specifies credentials to authorize access to Azure Blob Storage. Use one of this, account key or SAS token. |
| `azure.account.name` | Name of the Azure account to use |
| `azure.credentials.sharedkey.account.key` | Account key that specifies credentials to authorize access to Azure Blob Storage. Use one of this, connection string or SAS token. |
| `azure.credentials.sharedkey.sas.token` | SAS token (account or service) that specifies credentials to authorize access to Azure Blob Storage. Use one of this, connection string or account key. |
| `azure.blob.container.name` | Name of the Azure Blob Storage container that contains the blob |
| `azure.blob.inputstream.chunk.size.max.bytes` | Maximum size in bytes of each chunk of data when reading the blob contents chunk by chunk. Default: 4194304 |
| `azure.blob.path.delimiter` | String that separates elements of the path to the blob file. Default: / |
| `azure.blob.list.marker` | Marker that specifies the beginning of the next page of a list of
Azure Blob Storage items to fetch from Azure. This property is used internally.
|
#### Amazon Redshift [#amazon-redshift]
| field | Description |
| ---------- | ------------------------- |
| `path` | name of the table to use. |
| `rsuser` | Redshift database user |
| `rspasswd` | Redshift user password |
| `rsurl` | Redshift connection url |
#### Spark Map Reduce [#spark-map-reduce]
| field | Description |
| ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `sparkmaster` | address of the spark’s cluster master |
| `spark.job.process.jvm.heap.size.min` | Mininum JVM size used for the spark Driver process launched by the Jetty Rest Server. Default : 1024m |
| `spark.job.process.jvm.heap.size.max` | Maximum JVM size used for the spark Driver process launched by the Jetty Rest Server. Default: 2048m |
| Any Spark parameters be used also like: `spark.executor.memory` or `spark.executor.cores` | To pass such parameters, prefix them with ‘sml.sparkenv.’ as in the following examples: - sml.sparkenv.spark.executor.cores - sml.sparkenv.spark.cores.max |
| `spark.automl.sample.random.seed` | If AutoML is used, one can set the randomizer seed that will be used to select a random sample of tuple from the data source to be used to bootstrap the AutoML environement. |
**Spark Map Reduce Data Source Type**
**SymetryML 6.3.0 and later:** Due to changes in the underlying JDK and Jetty server, only Spark 4.1.0 is supported.
| Data Source | Spark 4.1.0 |
| ------------------------ | -------------------------------- |
| **Oracle OCIs3** | sparkocis3\_map\_reduce\_4\_1\_0 |
| **Amazon S3** | sparks3\_map\_reduce\_4\_1\_0 |
| **Google Cloud Storage** | sparkgcs\_map\_reduce\_4\_1\_0 |
| **Microsoft Azure Blob** | sparkabs\_map\_reduce\_4\_1\_0 |
#### JDBC [#jdbc]
| field | Description |
| -------- | ----------- |
| driver | |
| host | |
| port | |
| database | |
| user | |
| password | |
#### Snowflake [#snowflake]
SymetryML supports connecting to Snowflake data warehouses using Snowflake's Snowpark Container Services (CP3O) for distributed processing.
| field | Required | Description |
| --------------------------------------------- | ------------ | -------------------------------------------------------------------------------------------------------------------- |
| `snowflake.account` | **Required** | Snowflake account identifier (e.g., `XYZZZZZ-ZBC00000`) |
| `snowflake.account.role` | **Required** | Role to connect to the Snowflake account with (e.g., `ACCOUNTADMIN`) |
| `snowflake.user` | **Required** | User to connect to the Snowflake account with (e.g., `zzzuser`) |
| `snowflake.auth.private.key.raw` | **Required** | String content of the private key to authorize the connection to the account |
| `snowflake.data.database` | **Required** | Name of the database containing the data |
| `snowflake.data.schema` | **Required** | Name of the schema containing the data |
| `snowflake.data.table` | **Required** | Name of the table containing the data |
| `snowflake.cp3o.compute.pool.instance.family` | **Required** | Compute pool instance family to use (e.g., `CPU_X64_S`) |
| `snowflake.cp3o.compute.pool.max.nodes` | **Required** | Maximum number of nodes in the compute pool (e.g., `3`) |
| `snowflake.cp3o.compute.pool.min.nodes` | **Required** | Minimum number of nodes in the compute pool (e.g., `1`) |
| `snowflake.cp3o.worker.total` | **Required** | Number of workers that share the update task (e.g., `3`) |
| `snowflake.cp3o.data.table.order.by.clause` | **Required** | SQL clause used to order the rows of the table so that the workers can partition it properly (e.g., `ROW_INDEX ASC`) |
| `snowflake.job.warehouse` | Optional | Snowflake warehouse to use for queries. **Default: `WH_NB`** |
| `snowflake.job.warehouse.size` | Optional | Size of the Snowflake warehouse to use for queries. **Default: `XSMALL`** |
| `snowflake.job.warehouse.auto.suspend` | Optional | Inactivity timeout in seconds of the Snowflake warehouse to use for queries. **Default: `120`** |
| `snowflake.job.role` | Optional | Snowflake role to use in processing the job. **Default: `C3PO_APP_ROLE`** |
| `snowflake.list.roles` | Optional | Boolean flag to show a list of Snowflake roles while browsing. **Default: `false`** |
| `snowflake.cp3o.data.frame.max.size` | Optional | Maximum size of a Data Frame to use while updating SML projects in processing the job. **Default: `16384`** |
#### Amazon EMR [#amazon-emr]
| field | Required | Description |
|---|
chunksize | Optional | SymetryML process the data chunk by chunk. This parameters specifies the chunk size. Default: 5000 |
emr.client.aws.region | Optional | AWS region of the EMR cluster. Default: us-east-1 |
emr.cluster.ec2.key.name | Required | EC2 key pair name for the cluster. |
emr.cluster.ec2.subnet.id | Optional | EC2 subnet id for the cluster. Default: null |
emr.cluster.instance.count | Required | # of EC2 instances in the EMR cluster. |
emr.cluster.instance.master.type | Required | Instance type of the master EC2 instance. |
emr.cluster.instance.slave.type | Required | Instance type of the slave EC2 instances. |
emr.cluster.log.storage.enable | Optional | Boolean enabling for storing the EMR logs. Default: false |
emr.cluster.log.storage.uri | Optional | URI of the EMR logs. Default: null |
emr.job.flow.role | Optional | EMR role for EC2 that is used by EC2 instances within the cluster. Default: AWS EMR_EC2_DefaultRole |
emr.s3.job.bucket.name | Required | S3 bucket that stores the files needed for the Spark cluster job; it can include the directory that stores the EMR logs. |
emr.service.role | Optional | Amazon EMR role, which defines the allowable actions for Amazon EMR. Default: AWS EMR_DefaultRole |
path | Required | Path of data source to process, can be a folder. This is the 'data path' without the 'bucket part' |
s3accessKey | Required | AWS access key |
s3bucket | Required | AWS S3 bucket where data resides |
s3marker | Required | Optional marker parameter indicating where in the S3 bucket to begin listing. The list will only include keys that occur lexicographically after the marker. |
s3secretKey | Required | AWS secret key |
sml.sparkenv.* | Required | Allows to specify any Apache Spark environment configuration like: spark.cores.max e.g. use:sml.sparkenv.spark.cores.max or spark.executor.memory e.g. use:sml.sparkenv.spark.executor.memory |
sparksymproject | Required | Name of the project. |
### Additional CSV Options [#additional-csv-options]
You can specify additional parameters that describe the ‘type of csv files’. You can also add the following parameters to a data source to change how SymetryML parses your data:
| Key | Description |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| **csv\_entry\_separator** | Specifies which character to use as the delimiter for each record for a given tuple. |
| **csv\_quote\_character** | Specifies the quote character. |
| **csv\_strict\_quotes** | Setting this option to true discards characters outside the quotes. If there are no quotes between delimiters, an empty string is generated. |
| **csv\_header\_missing** | Specifies that this data source does not have any header. SymetryML can then generate a header automatically. |
### Additional Information on Spark S3 Data Source [#additional-information-on-spark-s3-data-source]
SymetryML can leverage a spark cluster to speed up processing of large amounts of data significantly. Currently, your data must reside on Amazon S3. Depending on the size of your data, it may take more or less time for the job to start, as the Spark Cluster must compute the partitions of your data before starting the job. Consequently, if your data is very large, this may take a few minutes.
Please be sure to consult the [Spark Installation Guide](/docs/guides/installation-guide/spark-installation-guide) for additional Spark information.
**Best practices for Spark S3 Data Source:**
* Performance may vary depending on Amazon resource utilization when you run your job.
* Be sure all executor nodes in your cluster reside in the same Amazon EC2 placement group.
### About Data Source Plugins (DSPlugins) [#about-data-source-plugins-dsplugins]
The SymetryML data source API allows you to create a new data source by the form of Java library (jar) that can be added the server. Instead of transforming data into CSV files, for example, you can write a DS plugin that reads the data natively.
## Data Source Encryption [#data-source-encryption]
Data sources might contain sensitive information that should never be passed in the clear. To avoid having to use HTTPS for these services, the SymetryML REST API forces you to pass such information in encrypted form. This can be done easily, as each SymetryML secret key is also a 128-bit Advanced Encryption Standard (AES) secret key.
1. Create your [DSInfo](/docs/symetryml-rest-client/rest-documentation/appendix-a-json-data-structure-schema#dsinfo-json) data structure and enter the appropriate information so that the SymetryML server can access it.
2. Extract the JSON string from that data structure.
3. Encrypt the JSON string representation using your SymetryML secret key:
* Initialization vector in Base 64: **LzM5QUtXZXWHm7HJ4wAePg==**
* Block cipher algorithm: **AES/CBC/PKCS5Padding**
4. Send the encrypted string as part of the body to any REST service that requests a [DSInfo](/docs/symetryml-rest-client/rest-documentation/appendix-a-json-data-structure-schema#dsinfo-json) as the body.
5. The server decrypts the string using the client secret key and reconstructs the [DSInfo](/docs/symetryml-rest-client/rest-documentation/appendix-a-json-data-structure-schema#dsinfo-json).
## Data Source Create [#data-source-create]
This API function creates a new data source.
### URL [#url]
```
POST /symetry/rest/{cid}/dss/create [Body=DSInfo (encrypted)]
```
### HTTP Responses [#http-responses]
| HTTP Status Code | HTTP Status Message | Description |
| ---------------- | ------------------- | ----------------------------------------------------- |
| **202** | CREATED | Success. |
| **409** | CONFLICT | A data source with the specified name already exists. |
### HTTP Response Entity [#http-response-entity]
None.
### Sample Request/Response [#sample-requestresponse]
Please note that in the following example the BODY is encrypted. Refer to section [Data Source Encryption](/docs/symetryml-rest-client/rest-documentation/data-source-api#data-source-encryption) for encryption details.
```
Request:
POST url="http://charm:8080/symetry/rest/c1/dss/create"
Body:
emUmJ7LXKaICPww/dKRoMR/Go6+B3ATTn+GwgE1/vcK9pN/mLoqGyKiMtGiTYMct4Gth1ElniKLXtmXfHfs7Rfn+QhJHZ+s00w2PBdbvYZIoF3My04H5XCboY21Fh4SkBhsxo+DhYuardN7R+uGphN/DSbiHRLIXe51HijSpfuq0fJuQYW0ccN4FM/B74LJccuDnbb+IouS9u/9rybKu/wsVbDVRTK/zZpWXyn4qtds=
Response:
{"statusCode":"CREATED","statusString":"DS created with name [ds1] for customer [c1]","values":{}}
```
## Data Source Update [#data-source-update]
This API function update an existing data source.
### URL [#url-1]
```
PUT /symetry/rest/{cid}/dss/{dsname} [Body=DSInfo (encrypted)]
```
### HTTP Responses [#http-responses-1]
| HTTP Status Code | HTTP Status Message | Description |
| ---------------- | ------------------- | ------------------------------------------------------ |
| **200** | OK | Success. |
| **404** | NOT FOUND | If data source with the specified name does not exist. |
### HTTP Response Entity [#http-response-entity-1]
None.
### Sample Request/Response [#sample-requestresponse-1]
Please note that in the following example the BODY is encrypted. Refer to section [Data Source Encryption](/docs/symetryml-rest-client/rest-documentation/data-source-api#data-source-encryption) for encryption details.
```
Request:
PUT url="http://charm:8080/symetry/rest/c1/dss/aDataSourceName"
Body:
emUmJ7LXKaICPww/dKRoMR/Go6+B3ATTn+GwgE1/vcK9pN/mLoqGyKiMtGiTYMct4Gth1ElniKLXtmXfHfs7Rfn+QhJHZ+s00w2PBdbvYZIoF3My04H5XCboY21Fh4SkBhsxo+DhYuardN7R+uGphN/DSbiHRLIXe51HijSpfuq0fJuQYW0ccN4FM/B74LJccuDnbb+IouS9u/9rybKu/wsVbDVRTK/zZpWXyn4qtds=
Response:
{"statusCode":"OK","statusString":"DS updated with name [ds1] for customer [c1]","values":{}}
```
## List Customer Data Sources [#list-customer-data-sources]
This API function returns all the data sources that belong to a user.
### URL [#url-2]
```
GET /symetry/rest/{cid}/dss/
```
### HTTP Responses [#http-responses-2]
| HTTP Status Code | HTTP Status Message | Description |
| ---------------- | ------------------- | ----------- |
| **200** | OK | Success. |
### HTTP Response Entity [#http-response-entity-2]
| HTTP Response Entity | Example |
| ----------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [**StringList**](/docs/symetryml-rest-client/rest-documentation/appendix-a-json-data-structure-schema#stringlist) | `{"statusCode":"OK","statusString":"OK","values":{"stringList":{"values":["Iris_SymetryML.csv-predict.csv:s3","h1:http","BigData11g_Test.csv:http","Iris_SymetryML.csv:s3","Smaato_Bids_20130812_CTR.csv:s3"]}}}` |
### Sample Request/Response [#sample-requestresponse-2]
```
Request:
GET url="http://charm:8080/symetry/rest/c1/dss"
Response:=
{"statusCode":"OK","statusString":"OK","values":{"stringList":{"values":["Iris_rtlm-out-neil.csv:s3","Iris_rtlm.csv:s3","BigData11g_num.csv:s3","Smaato_Bids_20130812_CTR.csv:s3"]}}}
```
## Delete Data Source [#delete-data-source]
This API function deletes a data source.
### URL [#url-3]
```
DELETE /symetry/rest/{cid}/dss/{dsname}
```
### HTTP Responses [#http-responses-3]
| HTTP Status Code | HTTP Status Message | Description |
| ---------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| **200** | OK | Success. |
| **409** | CONFLICT | Data source cannot be deleted. A SymetryML project might be using the data source. The response contains an error string with further details. |
### HTTP Response Entity [#http-response-entity-3]
None
## Data Source Information [#data-source-information]
This API function returns information about a data source. Because this information can be sensitive, the server encrypts it to ensure that the information will be returned safely, even when using HTTP. Use your SymetryML secret key to decrypt the information. For information about how to decrypt the response, see the section [Data Source Encryption](/docs/symetryml-rest-client/rest-documentation/data-source-api#data-source-encryption).
### URL [#url-4]
```
GET /symetry/rest/{cid}/dss/{dsname}
```
### HTTP Responses [#http-responses-4]
| HTTP Status Code | HTTP Status Message | Description |
| ---------------- | ------------------- | ----------- |
| **200** | OK | Success. |
### HTTP Response Entity [#http-response-entity-4]
| HTTP Response Entity | Example |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| **DSInfo (encrypted)** | [DSInfo](/docs/symetryml-rest-client/rest-documentation/appendix-a-json-data-structure-schema#dsinfo-json) encrypted using the customer secret key. |
### Sample Request/Response [#sample-requestresponse-3]
```
Request:
GET url="http://charm:8080/symetry/rest/c1/dss/BigData11g_num.csv"
Response:
{"statusCode":"OK","statusString":"OK","values":{"dsinfo":"emUmJ7LXKaICPww/dKRoMRNJEMepL37Eq9CgZfhZPWj93mo3A+C8ucfIOGaPwwn2dip/JEuLFjUT/fjHjy18XKFnzFz5Ujp0WmS0uA4ssvAJwNPL6BvnsY6+a/lKa+c/q9/5tz5lr13N13I7OGAhuYhXYV+xb8oFZqsn+bH5spBXRb5u+oyEMXNKLCaNt3pzc/xCyW47KCwIi9V5iSA+fcJAWfetm9ZsIHNbI6utkxKrqrU5OfLmgriGAP++yQtutlGR7r/bKV1bRc8UDsgsXQg1HgoxHcKXCgsAFDFzqsJmZ/5/uQDc0ytc5Fk85GUx"}}
```
## Data Source Browsing [#data-source-browsing]
This API function lists content about a remote data source directory.
### URL [#url-5]
```
POST /symetry/rest/{cid}/projects/dsbrowse/ [Body=DSListingRequest (encrypted)]
```
### HTTP Responses [#http-responses-5]
| HTTP Status Code | HTTP Status Message | Description |
| ---------------- | ------------------- | ----------- |
| **200** | OK | Success. |
### HTTP Response Entity [#http-response-entity-5]
| HTTP Response Entity | Description |
| ------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------- |
| [**DSListingResponse**](/docs/symetryml-rest-client/rest-documentation/appendix-a-json-data-structure-schema#dslistingresponse-json) | Contains listing information about the requested directory or folder. |
### Sample Request/Response [#sample-requestresponse-4]
```
Request:
POST url="http://charm:8080/symetry/rest/c1/projects/dsbrowse"
Body:
drhSjndw6G15pgevCsDqaSfjX9x3hMo+dNqd/MV943Dsd2rl2guhvq2qUhjEORcfKAEjHaoRZMKmSbQB6bcca2YT6HmUyRxuOG0wiKgGy0MOEq7+iIncbX4orpGr4rhro1Frw909Uy8qcWskaInQHJT4EGRPcwxvwInFlea39hsMkycFK4pKlTpanOUYgcv7
Response:
{"statusCode":"OK","statusString":"OK","values":{"dsdirectoryListing":{"ok":true,"dirs":["data2/","folder-a/","folder-datasets/","folder-demo/","folder-dev/","folder-dev-pub-http/","folder-dev-pub-https/","folder-docs/","folder-perf-all-reports/","folder-source/"]}}}
```
## Fetching Sample Data 1 [#fetching-sample-data-1]
This API function fetches a data source sample by specifying the data source information as part of the request body. The response returns up to 128 lines. This REST call needs a request body that contains an encrypted DSInfo data structure (see the section [Data Source Encryption](/docs/symetryml-rest-client/rest-documentation/data-source-api#data-source-encryption)).
### URL [#url-6]
```
POST /symetry/rest/{cid}/dss/sample/preview [body=DSInfo encrypted]
```
### HTTP Responses [#http-responses-6]
| HTTP Status Code | HTTP Status Message | Description |
| ---------------- | ------------------- | ----------- |
| **200** | OK | Success. |
### HTTP Response Entity [#http-response-entity-6]
| HTTP Response Entity | Description |
| -------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
| [**DataFrame**](/docs/symetryml-rest-client/rest-documentation/appendix-a-json-data-structure-schema#dataframe-json) | Contains a sample of the data source up to 128 lines. |
### Sample Request/Response [#sample-requestresponse-5]
```
Request:
POST url="http://charm:8080/symetry/rest/c1/dss/sample/preview
Body:
WTqUHBoXHbl+cMMjdc0zgjBP8e44G1os15V+I4GZgDOr1dX9uOfvY5uK9ZgC9yral9XC1ohD1W+UvkPlKR4dQT00EgCdS2UPgZz2NwwooHOM+KY1Ysf5qZlkFKiOkxwoWH/mr3mvvgdTUpZS8zrDJk3gwsavFT5fe0J2lTR33F1OH7FwxP4qs5nzRbVz546l
Response:
{"statusCode":"OK","statusString":"OK","values":{"dataframe":{"attributeNames":["adexchange","imp_width","imp_height","imp_btype","preference","pub","domain","site_base_url","category","device_ip","device_country","device_dma","device_state","device_city","zip","device_carrier","language","device_os","device_make","device_model","device_osv","lat","lon","restriction_bcat","restriction_badv","position","gender","user_keyword","user_yob","user_age","view_count","campaign","creative","creative_type","winner","bid_price","win_price","clicked","rtlm_ctr_score","rtdm","req_hour","req_day","category_count","bcat_count","badv_count"],"data":[["Smaato","320","50","","APP","New IT Solutions Ltd","","","Technology \u0026 Computing","157.55.32.83","US","","WASHINGTON","REDMOND","98052","","","Unknown","","","","47.67398834","-122.1215134","IAB7-28|IAB19-30|IAB22-1|IAB19-3|IAB17-18|IAB26|IAB25|IAB24|IAB9-9|IAB7-
(...)
,"attributeTypes":["S","C","C","L","S","L","S","S","S","S","S","S","S","S","S","S","S","S","S","S","S","S","S","L","S","S","S","S","S","S","B","S","S","S","B","C","C","B","S","B","C","C","B","C","B"]}}}
```
## Fetching Sample Data 2 [#fetching-sample-data-2]
### URL [#url-7]
```
GET /symetry/rest/{cid}/dss/{dsname}/sample
```
### HTTP Responses [#http-responses-7]
| HTTP Status Code | HTTP Status Message | Description |
| ---------------- | ------------------- | ----------- |
| **200** | OK | Success. |
### HTTP Response Entity [#http-response-entity-7]
| HTTP Response Entity | Description |
| -------------------- | --------------------------------------------------------------------------------------------------------------------- |
| **DataFrame** | See [DataFrame](/docs/symetryml-rest-client/rest-documentation/appendix-a-json-data-structure-schema#dataframe-json). |
### Sample Request/Response [#sample-requestresponse-6]
```
Request:
GET url="http://charm:8080/symetry/rest/c1/dss/Smaato_Bids_20130812_CTR.csv/sample"
Response:
{"statusCode":"OK","statusString":"OK","values":{"dataframe":{"attributeNames":["adexchange","imp_width","imp_height","imp_btype","preference","pub","domain","site_base_url","category","device_ip","device_country","device_dma","device_state","device_city","zip","device_carrier","language","device_os","device_make","device_model","device_osv","lat","lon","restriction_bcat","restriction_badv","position","gender","user_keyword","user_yob","user_age","view_count","campaign","creative","creative_type","winner","bid_price","win_price","clicked","rtlm_ctr_score","rtdm","req_hour","req_day","category_count","bcat_count","badv_count"],"data":[["Smaato","320","50","","APP","New IT Solutions Ltd","","","Technology \u0026 Computing","157.55.32.83","US","","WASHINGTON","REDMOND","98052","","","Unknown","","","","47.67398834","-122.1215134","IAB7-28|IAB19-30|IAB22-1|IAB19-3|IAB17-18|IAB26|IAB25|IAB24|IAB9-9|IAB7-
(...)
,"attributeTypes":["S","C","C","L","S","L","S","S","S","S","S","S","S","S","S","S","S","S","S","S","S","S","S","L","S","S","S","S","S","S","B","S","S","S","B","C","C","B","S","B","C","C","B","C","B"]}}}
```
## SymetryML Project Data Source API [#symetryml-project-data-source-api]
### Add Data Source to a SymetryML project [#add-data-source-to-a-symetryml-project]
This API function lets you add a data source to a project.
#### URL [#url-8]
```
GET /symetry/rest/{cid}/projects/{pid}/addds/{dsname}
```
#### HTTP Responses [#http-responses-8]
| HTTP Status Code | HTTP Status Message | Description |
| ---------------- | ------------------- | ----------- |
| **200** | OK | Success. |
#### HTTP Response Entity [#http-response-entity-8]
None
### Remove Data Source from a SymetryML project [#remove-data-source-from-a-symetryml-project]
This API function lets you remove a data source from a project.
#### URL [#url-9]
```
GET /symetry/rest/{cid}/projects/{pid}/detachds/{dsname}
```
#### HTTP Responses [#http-responses-9]
| HTTP Status Code | HTTP Status Message | Description |
| ---------------- | ------------------- | ----------- |
| **200** | OK | Success. |
#### HTTP Response Entity [#http-response-entity-9]
None
### Learning Data from a Data Source [#learning-data-from-a-data-source]
This API function lets you learn from a previously created data source.
#### URL [#url-10]
```
GET /symetry/rest/{cid}/projects/{pid}/dss/{dsname}/learn
```
#### HTTP Responses [#http-responses-10]
| HTTP Status Code | HTTP Status Message | Description |
| ---------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **200** | OK | Success. Includes an HTTP Location header specifying the location of the job ID that was created to handle the request. Example: `{"statusCode":"ACCEPTED","statusString":"Job Created","values":{}}` |
#### HTTP Response Entity [#http-response-entity-10]
None
#### Sample Request/Response [#sample-requestresponse-7]
```
Request 1 create ds:
POST url=http://charm:8080/symetry/rest/c1/dss/create
Body 1:
{ENCRYPTED}
Request 2 learn ds:
POST url=http://charm:8080/symetry/rest/c1/projects/r1/dss/irisds/learn
BODY 2
{"errorHandling":1,"attributeNames":["sepal_length","sepal_width","petal_length","petal_width","sepal_lengt_b1","sepal_lengt_b2","sepal_width_b1","sepal_width_b2","petal_length_b1","petal_length_b2","petal_width_b1","petal_width_b2","Iris_setosa","Iris_versicolor","Iris_virginica"],"data":[],"attributeTypes":["C","C","C","C","B","B","B","B","B","B","B","B","B","B","B"]}
Response 2:
{"statusCode":"ACCEPTED","statusString":"Job Created","values":{}}
Response 2 Header:
Location: http://charm:8080/symetry/rest/c1/jobs/4
Job 3 Request:
GET url="http://charm:8080/symetry/rest/c1/jobs/4
Job 3 Response:
{"statusCode":"OK","statusString":"Job is finished","values":{"smlInfo":{"pid":"r1","isDirty":true,"modelsList":[],"modelTypeList":[],"attributeNames":["sepal_length","sepal_width","petal_length","petal_width","sepal_lengt_b1","sepal_lengt_b2","sepal_width_b1","sepal_width_b2","petal_length_b1","petal_length_b2","petal_width_b1","petal_width_b2","Iris_setosa","Iris_versicolor","Iris_virginica"],"attributeIndexes":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14],"attributeTypes":["C","C","C","C","B","B","B","B","B","B","B","B","B","B","B"],"modelAssessments":{},"modelPredictions":{},"hash":-1,"categorySeparator":"$","type":"cpu","creationDate":1488220762857,"lastModificationDate":1488226030143,"loaded":true,"persisted":true}}}
```
### Forgetting Data from a Data Source [#forgetting-data-from-a-data-source]
This API function lets you forget data from a previously created data source.
#### URL [#url-11]
```
GET /symetry/rest/{cid}/projects/{pid}/dss/{dsname}/forget
```
#### HTTP Responses [#http-responses-11]
| HTTP Status Code | HTTP Status Message | Description |
| ---------------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **200** | OK | Success. Includes an HTTP Location header specifying the location of the job ID that was created to handle the request. For example: `{"statusCode":"ACCEPTED","statusString":"Job Created","values":{}}` |
#### HTTP Response Entity [#http-response-entity-11]
None
#### Sample Request/Response [#sample-requestresponse-8]
See the section previous [Learn](/docs/symetryml-rest-client/rest-documentation/data-source-api#learning-data-from-a-data-source) endpoint and replace `learn` with `forget` in the url.
### Prediction Based on a Data Source [#prediction-based-on-a-data-source]
After a model is built, you can use this API function to make predictions using a data source. This action can be performed on very large files if they reside on Amazon S3. A prediction file is created that contains the original rows, along with additional prediction information based on the type of model used.
#### URL [#url-12]
```
Request:
POST /symetry/rest/{cid}/projects/{pid}/dss/predict/{modelid}?indsname={indsname}&outdsname={outdsname}&impute=false
Request Body:
{
"attributeNames":[{input attributes names}],
"attributeTypes":[{input attribute types}]
}
```
#### Query Parameters [#query-parameters]
| Parameter | Required / Optional | Description |
| ------------- | ------------------- | -------------------------------------------------------------- |
| **indsname** | Required | Data source to use as input file for prediction. |
| **outdsname** | Required | Data source to use as output file for prediction. |
| **impute** | Optional | Boolean parameter specifying whether to impute missing values. |
#### HTTP Responses [#http-responses-12]
| HTTP Status Code | HTTP Status Message | Description |
| ---------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| **202** | ACCEPTED | Success. Includes an HTTP Location header specifying the location of the job ID that was created to handle the request. |
| **500** | INTERNAL SERVER ERROR | If the server refuses to accept the new job, it notifies the client with the error "Job execution was refused by server." |
#### HTTP Response Entity [#http-response-entity-12]
None
### Sample Request/Response [#sample-requestresponse-9]
```
Request:
POST url="http://charm:8080/symetry/rest/c1/projects/irisTest/dss/predict/testLDA?indsname=dsin&outdsname=dsout&impute=false
Request Body:
{
"attributeNames":["sepal_length","sepal_width","petal_length","petal_width"],
"attributeTypes":["C","C","C","C"]
}
Response Header:
Location: http://charm:8080/symetry/rest/c1/jobs/2
Response:
{"statusCode":"ACCEPTED","statusString":"Job Created","values":{}}
Job Request:
GET url="http://charm:8080/symetry/rest/c1/jobs/2
Job Response:
{"statusCode":"OK","statusString":"Job is finished","values":{"dataframe":{"attributeNames":["resZ_Result","res_Result","normZ_Result","sepal_length","sepal_width","petal_length","petal_width","sepal_lengt_b1","sepal_lengt_b2","sepal_width_b1","sepal_width_b2","petal_length_b1","petal_length_b2","petal_width_b1","petal_width_b2","Iris_setosa","Iris_versicolor","Iris_virginica"],"data":[["-0.4559922893809558","1","0.2929126883636288","4.3","3","1.1","0.1","1","0","0","1","1","0","1","0","1","0","0"],["-0.5225734165408404","1","0.2965158290613048","
(…)
["-2.6526076600477295","1","0.4232667751179995","6.9","3.1","5.4","2.1","0","1","0","1","0","1","0","1","0","0","1"]]}}}
```
## Encoder Data Source API [#encoder-data-source-api]
### Updating an Encoder with a Data Source [#updating-an-encoder-with-a-data-source]
This API function updates an Encoder with data from a data source.
#### URL [#url-13]
```
GET /symetry/rest/{cid}/encoders/{encodername}/learnds [body = DSInfo (encrypted)]
```
#### HTTP Responses [#http-responses-13]
| HTTP Status Code | HTTP Status Message | Description |
| ---------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| **202** | ACCEPTED | Success. Includes an HTTP Location header specifying the location of the job ID that was created to handle the request. |
| **500** | INTERNAL SERVER ERROR | If the server refuses to accept the new job, it notifies the client with the error "Job execution was refused by server." |
#### HTTP Response Entity [#http-response-entity-13]
None
### Sample Request/Response [#sample-requestresponse-10]
```
Request:
POST url="http://charm:8080/symetry/rest/c1/encoders/enctest/learnds
Request Body:
emUmJ7LXKaICPww/dKRoMZo4y++JZksKFU3xx80mZ0E83ED6eCXhZ7PR1jzizIQg/uH19jVkSUWnDxul9+NIqZe7a9Yz5a8gW1kpRHJ7SyOYoo7oV90atrkMcZ73Jj0FRn53P81t8Q+7fCTwMYRfD52hjCbwtvdwFCbPbhPo1c9CAk9QLhtAFLDWVqblBeBXTPR8/0zfKJWtwj30Yr0gwqeTI3+BMOMEvH28WFWXo+wBdgkMBpXJsQH/zbPvVdCp9P+BJWp/E1Ju2hUzPO5c2k3/Dmqv3xWNmsQzEJWNMFTJnBG33hTUyTr/+j87NsM2e1luWvf1KaNzaatjS1ZZ9AQCp7gv48vMVrzEHs4ePiOFq5t0UElAu1kzFerhtNxEFmC9A90Gjt3FSLrEGgx5emZ0uogJa6m4nufbQnKUrUZL1sILmLfReZOqnPKp59HqjiiczpkPQ2vPfhmFpaM2RYaEHbqSpMGCnBO9Axij9ExPjR4X9aqHOzKw2yBba4Em
Response Header:
Location: http://charm:8080/symetry/rest/c1/jobs/4
Response:
{"statusCode":"ACCEPTED","statusString":"Job Created","values":{}}
Job Request:
GET url="http://charm:8080/symetry/rest/c1/jobs/4
Job Response:
{"statusCode":"OK","statusString":"Job is finished","values":{}}
```
### Listing a Data Source Used by an Encoder [#listing-a-data-source-used-by-an-encoder]
This API function lists the data source(s) that were used to update an Encoder.
#### URL [#url-14]
```
GET /symetry/rest/{cid}/encoders/{encodername}/dss
```
#### HTTP Responses [#http-responses-14]
| HTTP Status Code | HTTP Status Message | Description |
| ---------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| **202** | ACCEPTED | Success. Includes an HTTP Location header specifying the location of the job ID that was created to handle the request. |
#### HTTP Response Entity [#http-response-entity-14]
| HTTP Response Entity | Example |
| -------------------- | ----------------------------------------------------------------------------------------------------------------- |
| **StringList** | See [StringList](/docs/symetryml-rest-client/rest-documentation/appendix-a-json-data-structure-schema#stringlist) |
#### Sample Request/Response [#sample-requestresponse-11]
```
Request:
GET url="http://charm:8080/symetry/rest/c1/encoders/enctest/dss
Response:
{"statusCode":"OK","statusString":"OK","values":{"stringList":{"values":["s12"]}}}
```
## Data Source Job Status [#data-source-job-status]
When invoking a JobStatus for a job that was initiated for learning, forgetting, or making predictions based on a data source, the response might contain an entity. The following sections describe these cases.
### Learning a Data Source [#learning-a-data-source]
#### HTTP Responses [#http-responses-15]
| HTTP Status Code | HTTP Status Message | Description |
| ---------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **200** | OK | No entity. |
| **202** | ACCEPTED | Job is not finished. Includes a [JobInfo](/docs/symetryml-rest-client/rest-documentation/appendix-a-json-data-structure-schema#jobinfo) entity. Using the current and maximum fields, you can compute the percentage of the job that was accomplished so far. "Maximum" refers to the size of the file to process and contains the approximate number of bytes processed by the job. |
### Forgetting a Data Source [#forgetting-a-data-source]
#### HTTP Responses [#http-responses-16]
| HTTP Status Code | HTTP Status Message | Description |
| ---------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **200** | OK | No entity. |
| **202** | ACCEPTED | Job is not finished. Includes a JobInfo entity. For more information, see the section [JobInfo](/docs/symetryml-rest-client/rest-documentation/appendix-a-json-data-structure-schema#jobinfo) and also the section on [Symetry Jobs](/docs/symetryml-rest-client/rest-documentation/symetryml-job-information). Using the current and maximum fields, you can compute the percentage of the job that was accomplished so far. "Maximum" refers to the size of the file to process and contains the approximate number of bytes processed by the job. |
### Prediction Based on Data Source [#prediction-based-on-data-source]
### HTTP Responses [#http-responses-17]
| HTTP Status Code | HTTP Status Message | Description |
| ---------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **200** | OK | DataFrame that contains a sample of the predictions (up to 128 lines). Because Amazon files can be very large, it is not possible to return the prediction result file in its entirety within a REST call. Use your favorite tool to fetch the prediction results from the data source (S3 or SFTP). The prediction result file contains all the original file columns, plus the additional prediction column for each row. Any additional columns depend on the type of model used to make the predictions. |
| **202** | ACCEPTED | Job is not finished. Includes a [JobInfo](/docs/symetryml-rest-client/rest-documentation/appendix-a-json-data-structure-schema#jobinfo) entity. Using the current and maximum fields, it is possible to compute the percentage of the job that was accomplished so far. "Maximum" refers to the size of the file to process and contains the approximate number of bytes processed by the job. |