SymetryML Release Notes

A stateful drift detector that is fit once against reference projects and then reused to score any number of projects, returning PCA reconstruction and Mahalanobis scores together. This complements the existing one-shot /projects/dataDrift endpoint, which recomputes from both projects on every call. See Drift Calculator API.

POST   /symetry/rest/{cid}/driftCalculator?id={id}
POST   /symetry/rest/{cid}/driftCalculator/{dcid}/fit
POST   /symetry/rest/{cid}/driftCalculator/{dcid}/calculate
GET    /symetry/rest/{cid}/driftCalculator/{dcid}/info
GET    /symetry/rest/{cid}/driftCalculators
DELETE /symetry/rest/{cid}/driftCalculator/{dcid}

Data Sources

Google Dataproc

A new dataproc data source type runs processing on a Google Cloud Dataproc cluster, reading data from Google Cloud Storage. Authentication resolves from the data source key, then a server setting, then Google Application Default Credentials — so on GCP no service-account JSON needs to be pasted in. Previewing a Dataproc data source reads GCS directly and does not launch a cluster. See Google Dataproc.

PostgreSQL and Amazon Redshift over JDBC

The jdbc data source type now supports the PostgreSQL family, selected by setting jdbcprefix to jdbc:postgresql or jdbc:redshift. The path takes the form database/table. Browsing lists databases through a maintenance connection and lists tables from the public schema. See PostgreSQL and Amazon Redshift over JDBC.

Snowflake Compute Pool Orchestrator

Snowpark Compute Pool processing is now enabled. In 6.3 a data source configured for CP3O was rejected at run time with Snowflake cluster is not available.

Field validation

Data source fields are now validated on create, update and sample preview, returning 400 BAD REQUEST with a message naming the missing field. In 6.3 these surfaced as generic 500 errors. Every data source type table now carries a Required column. See Field Validation.

HTTP and HTTPS data sources

Rewritten onto the JDK HTTP client. Redirects are now followed, a non-2xx response fails with HTTP GET <url> failed with status N, and a server that sends no Content-Length results in a reported size of -1.

Exploration

Per-request output precision

/explore takes a new precision query parameter — an integer in [0, 17], default 4. It replaces the server-wide sml.explore.double.format setting, which has been removed.

Partial failures

A bad MLContext no longer fails the whole /explore request. Each MLContext is evaluated independently and a failure becomes a map whose single key is the error message with value -1.0, inside a 200 OK. See Partial Failures.

Univariate quantiles

The uni metric accepts uni_q_<percentile> keys in extraParameters and returns a matching qtl_<percentile> entry.

useLocal on PCA, SVD, SVD feature selection and VIF

These four endpoints now accept the useLocal query parameter, matching /explore.

Prediction

Multiple models in one learnpredict call

The {modelid} path segment accepts a comma-separated list of model ids. Each model contributes its own columns to the returned DataFrame, named <model id>::<model key>.

Cross-Cutting Changes

Request body size limit

Request bodies are capped at 256 MB by default, configurable with request.body.max.size. A larger body is rejected with 413 Payload Too Large. See Request Body Size Limit.

CORS

The REST API can now emit CORS headers, letting browser pages call it directly. Off by default; enabled with -Dsml.rest.cors=true. See Cross-Origin Resource Sharing (CORS).

Improvements

JDBC and Snowflake values

Values from JDBC and Snowflake data sources are now carried through directly instead of passing through an intermediate CSV step. Column names and values that contain commas, quotes or newlines are handled correctly.

If you previously loaded JDBC or Snowflake data whose column names or values contain commas, quotes or newlines, re-load it to pick up this improvement.

JDBC NULL handling

A SQL NULL now maps to the empty string, the same missing-value convention used elsewhere in SymetryML. Snowflake already behaved this way.

Federated exploration

Bivariate exploration, density estimates and conditional density estimates on a Federated project now use the correct project for each MLContext in a multi-context request.

Removed

Legacy Spark data source types. Only spark<store>_map_reduce_4_1_0 remains; the unversioned sparks3_map_reduce and the 2.4.x / 3.0.x variants are rejected with Invalid ds type.
sml.explore.double.format. Replaced by the per-request precision parameter.

Upgrade Notes

Exploration results are no longer truncated to four decimals. Output now carries full precision unless precision is supplied. Any client parsing the old four-decimal output will see more digits.
A 200 OK from /explore no longer means every MLContext succeeded. Check each returned map for a single entry whose value is -1.0.
Request bodies larger than 256 MB now fail with 413. Raise request.body.max.size, or set it to 0 or less to disable the cap.
Data sources missing a required field now fail at create time with a 400 instead of failing later.
Re-load any JDBC or Snowflake data containing commas, quotes or newlines in column names or values — see Improvements above.
Data sources using a legacy Spark type must be recreated with the _4_1_0 equivalent.

Version 6.3

New REST API Endpoints

LearnPredict

Combines learn and predict operations in a single call, enabling real-time learn-then-predict workflows without separate requests.

POST /symetry/rest/{cid}/projects/{pid}/learnpredict/{modelid}

Performance Estimation

Estimates supervised model performance on production data when ground truth labels are not yet available. Model-agnostic and label-free, it supports any binary classification model and estimates confusion matrix-based metrics (ROC curve, precision-recall curve, precision, recall, accuracy, F1 score, specificity). Based on the Probabilistic Adaptive Performance Estimation (PAPE) framework with a SymetryML-optimized single-pass implementation for streaming environments.

POST /symetry/rest/{cid}/projects/performanceEstimator

Data Drift Detection

Detects data drift between a baseline project and an analysis project using multiple drift metrics: PCA reconstruction error, global drift, marginal statistical, and marginal non-statistical methods. Integrates with Rolling Window projects for continuous drift monitoring.

POST /symetry/rest/{cid}/projects/dataDrift

EVT Wrapper Management

Creates or deletes Extreme Value Theory (EVT) wrappers on any model for dynamic anomaly thresholding. Instead of static thresholds, EVT wrappers track prediction outputs and use Extreme Value Theory to dynamically flag anomalies.

POST /symetry/rest/{cid}/projects/{pid}/{modelid}/evtwrapper
DELETE /symetry/rest/{cid}/projects/{pid}/{modelid}/evtwrapper

Conditional Density Estimates

Computes conditional density estimates (histograms) for pairs of attributes, useful for bivariate analysis with conditional distributions. Supports federated projects via the useLocal parameter.

POST /symetry/rest/{cid}/projects/{pid}/conditionalDensityEstimate

Survival Analysis

Cumulative Incidence Function (CIF)

New survival analysis model for competing risks. Unlike standard survival methods that can overestimate event probabilities by treating competing events as censored, CIF properly accounts for the fact that experiencing one event precludes another.

Get Total Events -- POST /{cid}/projects/{pid}/cif/NtotEvents
Predict At Risk -- POST /{cid}/projects/{pid}/cif/predictAtRiskCIF
Predict Event -- POST /{cid}/projects/{pid}/cif/predictEvent

CIF project configuration parameters: cif_time_column, cif_event_column, cif_group_column, cif_event_of_interest, and sml_surv_type (set to cif).

Kaplan-Meier REST Endpoints

New REST endpoints for querying Kaplan-Meier survival models, including survival data frame retrieval and survival predictions with confidence intervals.

Get Survival DataFrames -- GET /{cid}/projects/{pid}/km/getSurvivalFrame
Kaplan-Meier Predict -- POST /{cid}/projects/{pid}/km/kaplanmeier

Federated Learning

Contract Management

New endpoints for managing data sharing contracts on federated projects:

Add Contract -- POST /{cid}/fedml/{pid}/addContract
Validate Contract -- POST /{cid}/fedml/{pid}/contractValidate

Federation Invitation

New alias endpoint for getting encrypted federation invitations:

POST /symetry/rest/{cid}/fedml/{pid}/getInvitation

The previous getEncrypted path is preserved for backward compatibility.

Exploration Enhancements

Kurtosis: New univariate statistic (kurtosis) available in exploration results.
Enhanced Bivariate Statistics: New metrics including conditional sum (condSum), conditional standard deviation (condStddev), and 95% confidence interval bounds (biCi95Down, biCi95Up).
Bivariate Conditional Flags: New parameters need_bi_conditional, need_bi_conditional_mmm, and need_bi_conditional_ci95 to request conditional statistics in bivariate exploration.
PCA Max Dimension: PCA results are now limited to 100 dimensions maximum by default. This value is configurable.
Explore Endpoint: New returnProjectInfo query parameter to control whether project info is included in the response.
Density Estimates: New useLocal parameter for federation support; improved threading.

Data Sources

Snowflake Data Source

New Snowflake data source type (snow) with support for:

Account, user, role, and private key authentication
Database, schema, and table configuration
Snowpark Container Services (CP3O) compute pool configuration
Job warehouse settings and data frame size limits

NATS Data Source

New NATS data source type (nats) with support for three data formats: Protobuf, JSON, and CSV. Protobuf format requires protobuf.schema and protobuf.msg.type.name parameters.

Spark 4.1.0

Spark support updated to version 4.1.0, required due to changes in the underlying JDK and Jetty server. Only Spark 4.1.0 is supported in this version.

Model Selection and AutoML

Genetic Algorithm Feature Selector (Experimental)

New evolutionary optimization approach for feature selection. Evolves a population of candidate feature sets over multiple generations using selection, crossover, and mutation to discover high-performing feature combinations. Use selector_type_genetic as the selector type.

Bayesian Optimization Feature Selector (Experimental)

New probabilistic modeling approach for efficient feature space search. Builds a surrogate model and uses an Upper Confidence Bound acquisition function to balance exploration and exploitation. Use selector_type_bayesian as the selector type.

MRMR Filter

New Minimum Redundancy Maximum Relevance filter (use_mrmr_filter) for feature pre-filtering during model selection.

Early Detection Assessment Metric

New early_detection assessment metric for binary classification models. Rewards models that return a positive prediction earlier in a sequence of observations. Designed for anomaly or malfunction detection on equipment where earlier detection yields a better score.

Feature Interaction Limit

New automl_feature_interaction_max_number parameter to control the maximum number of feature interactions during AutoML.

Rolling Window Projects

New project configuration for retaining statistics from only the most recent N rows of learned data:

sml_project_rolling_window_size -- specifies the rolling window length
sml_project_rolling_window_use_embedded -- creates an embedded project inside the main project for the rolling window
The embedded project can be referenced in the Data Drift API using rolling_window_embedded_project as the analysis project name

Infrastructure Changes

Logging Framework: Migrated from Log4j 1.x to Log4j 2.x
Jetty Upgrade: Updated to Jetty 12.x (Jakarta EE)
Explore Endpoint: Updated with returnProjectInfo parameter
Density Estimates: Updated with federation support and improved threading

Version 6.2

New Features

SSO Authentication Support

SymetryML now supports Single Sign-On (SSO) authentication via OIDC (OpenID Connect) and LDAP. After SSO authentication, SymetryML credentials (username and secret key) are returned for API access. SSO can also auto-create SymetryML user accounts based on the SSO profile.

Server Logs API

A new REST endpoint allows retrieval of the last N lines from the server log file. The number of lines returned is controlled by the SML_LOG_TAIL_SIZE configuration parameter.

GET /symetry/rest/{cid}/logs

Infrastructure Changes

Jakarta EE Migration: Migrated from javax.ws.rs to jakarta.ws.rs (JDK 11 support)
All existing REST endpoints maintain full backward compatibility

SymetryML Release Notes

On this page