Technologies Supported
DataKnits supports connections to 40+ technologies across 8 categories. Each technology has a dedicated reference page covering all connection fields, field dependencies, authentication methods, and any known limitations.
Relational Databases (JDBC)
These connectors use JDBC to communicate with on-premise and cloud-managed relational databases. They support metadata browsing (schemas, tables, columns), data preview, and both read and write operations in pipelines.
On-premises and cloud-hosted (RDS, Cloud SQL, Azure Database for PostgreSQL). Supports SSL, SSH tunnel, and IAM authentication for RDS.
On-premises and cloud-managed (RDS, Cloud SQL). Supports SSL mode, SSH tunneling, and timezone configuration.
On-premises Oracle 12c+, Oracle RDS, and Oracle Autonomous Database. Supports Wallet-based authentication for Autonomous DB.
On-premises SQL Server 2016+ and Azure SQL / SQL Managed Instance. Supports Windows Authentication passthrough and Azure AD auth.
Enterprise Teradata systems. Uses JDBC for metadata and preview. Supports TPT (Teradata Parallel Transporter) for high-volume bulk loading in generated code.
On-premises IBM Db2 LUW. JDBC-based connectivity with SSL support.
Wire-compatible with PostgreSQL — connects through the same JDBC PostgreSQL plugin. Requires a private IP reachable from inside the AlloyDB cluster's VPC.
Cloud Data Warehouses
Analytical and OLAP systems optimised for large-scale query workloads. DataKnits generates native bulk-load SQL (e.g., Snowflake COPY INTO, Redshift COPY) for these targets rather than row-by-row JDBC inserts, resulting in significantly faster data loads.
Supports username/password, RSA key pair, and OAuth token authentication. Configurable warehouse, database, schema, and role. Generated code uses the Snowflake Spark Connector for bulk operations.
Delta Lake on Azure or AWS. Authenticates via Personal Access Token. Supports direct path access (ABFSS/S3A) and Databricks-mounted storage (/mnt/). Unity Catalog integration available.
Service Account JSON key or Workload Identity. Generated code uses the BigQuery Spark Connector with GCS as intermediate storage for high-speed writes.
Username/password or IAM authentication. For large S3-to-Redshift loads, DataKnits generates the native Redshift COPY command using parallel ingest into distribution slices.
Service Principal or Managed Identity. Uses PolyBase or the COPY statement to load from ADLS Gen2 into Synapse distributed tables.
Wallet-based authentication (Oracle Cloud Wallet ZIP). Supports DBMS_CLOUD package for direct OCI Object Storage ingestion.
Cloud Object Storage
Object storage systems are used as both data sources (reading files) and data targets (writing files and staging data for warehouse loads). DataKnits uses Hadoop-compatible filesystem connectors (S3A, GCS, ABFS) — no cloud vendor SDKs are imported by the DataKnits backend.
Supports Static Credentials, IAM Role (STS AssumeRole), EC2/EKS Instance Profile, and S3-compatible endpoints (e.g., MinIO). Path-style access configurable for non-AWS stores.
Service Principal, Managed Identity (system or user-assigned), Storage Account Key, or SAS Token. Supports both WASB (Blob) and ABFSS (ADLS Gen2) protocols.
Service Account JSON Key or Workload Identity. Generated code uses the GCS Hadoop connector (gs://) for all read and write operations.
S3-compatible endpoint using API keys or Instance Principal. Compatible with the S3A Hadoop connector. Also supports oci:// native protocol where available.
File Transfer & Local Storage
File-based sources including SFTP, FTP/FTPS, and network shares are first-class connectors in DataKnits. They support the same metadata browsing, data preview, and pipeline integration as cloud storage. For on-premises file servers, the On-Prem Agent may be required.
Port 22, SSH. Supports password or SSH private key (PEM) authentication. Generates PySpark code that streams directly from SFTP into memory — no data is written to the DataKnits server disk.
Standard FTP (port 21) and FTPS (explicit or implicit TLS). Username/password authentication. Supports passive and active mode.
Windows and Linux network file shares. SMB requires username, password, and domain. NFS uses mount point paths. Typically requires the On-Prem Agent for access from the DataKnits control plane.
Streaming & Messaging
Streaming connectors have fundamentally different semantics to batch connectors — data is unbounded, offset-based, and schema-on-arrival. Streaming pipelines use dedicated node types in the pipeline canvas and generate Spark Structured Streaming code.
Bootstrap servers, topic configuration, consumer group ID. Supports PLAINTEXT, SASL_PLAINTEXT (PLAIN, SCRAM-256, SCRAM-512), SASL_SSL, and mTLS security protocols.
Managed Kafka services. MSK supports IAM or SCRAM authentication. Confluent Cloud uses API Key + Secret. Same pipeline node as on-premises Kafka.
Connection string or Service Principal authentication. Exposed as a Kafka-compatible endpoint — uses the same Kafka connector with Event Hubs bootstrap URL and SASL/OAUTHBEARER.
Static credentials or IAM Role. Stream name, AWS region, and shard iterator configuration. Generated code uses the Kinesis Spark connector for structured streaming.
Service Account Key or Workload Identity. Subscription-based consumption. Generated code uses the Spark–Pub/Sub connector.
Queue Manager, Channel, Host, Port (default 1414). Supports point-to-point (PUT/GET) and Pub/Sub via topics. Uses IBM MQ REST Messaging API (mqweb). Note: transactional GET (MQBEGIN/MQBACK) is not yet supported — each GET is immediately destructive.
Service URL, topic, and subscription configuration. Supports token-based and TLS client authentication.
NoSQL & Document Stores
NoSQL connectors allow DataKnits to read from and write to document, wide-column, and key-value stores as part of a pipeline. Schema is inferred at connection time where the store supports it.
Connection string or host/port/auth-db. Supports username/password and X.509 certificate authentication. Atlas and on-premises supported.
Contact points, port, datacenter, keyspace, and credentials. DataStax Astra uses a Secure Connect Bundle (ZIP) for cloud connectivity.
AWS region, table name, Static Credentials, or IAM Role. Read operations export DynamoDB items as a flattened DataFrame.
Endpoint, primary key or Service Principal. Supports SQL (Core), Cassandra, and MongoDB API modes. Container throughput is not managed by DataKnits.
Host URL, index name, API Key or username/password. Documents are flattened to a DataFrame using schema inference on the first N records.
Host, port, and optional password/TLS. Read-only — used for reference data lookups (e.g., enriching a pipeline with cached dimension data). Write to Redis is not supported.
Wide-column store, one connection per instance. Service Account Key or Workload Identity. No fixed schema — metadata browsing surfaces column families only. Read/write via Spark/Dataproc; no ETL-Native row executor yet.
Lakehouse & Data Catalog
Lakehouse catalog integrations allow DataKnits to treat Delta, Iceberg, and Hudi tables as first-class metadata entities — browsable, previewable, and usable as pipeline sources or targets without specifying raw file paths.
On S3, Azure Blob/ADLS Gen2, or GCS. Supports Time Travel (read by version or timestamp), schema enforcement, and Optimize/Vacuum operations. Databricks and open-source Delta supported.
REST Catalog, Glue Catalog, Hive Metastore, or Nessie. Supports table versioning, time travel, and schema evolution. On-premises Iceberg Lakehouse deployable via Docker Compose.
IAM Role or Static Credentials, AWS region. Acts as a metastore for S3-backed tables. DataKnits reads table definitions from Glue and generates PySpark code that queries tables by their Glue catalog name.
Workspace URL and Personal Access Token. DataKnits browses three-level namespaces (catalog.schema.table) and generates code using Unity Catalog table references directly.
HiveServer2 host, port, and database. Username/password or Kerberos. Used as both a metastore catalog and as a query execution target via JDBC.
Supported File Formats
These formats apply to any file-based source or target — cloud object storage (S3, GCS, Azure), SFTP/FTP, network shares, or local filesystem. Format selection is made per-node in the pipeline canvas.
| Format | Read | Write | Notes |
|---|---|---|---|
| CSV | ✅ | ✅ | Configurable delimiter (, | ; \t), quote character, multi-line records, header row detection, schema inference. |
| Parquet | ✅ | ✅ | Columnar format. Supports predicate pushdown (reading only required rows and columns). Recommended format for performance-sensitive pipelines. |
| ORC | ✅ | ✅ | Optimised for Hive and Hadoop workloads. Better compression than Parquet for text-heavy data. |
| JSON | ✅ | ✅ | Handles nested objects and arrays. Nested structures are expanded into columns. JSON Lines (one record per line) supported for large-scale ingestion. |
| Avro | ✅ | ✅ | Schema-first serialisation format. Schema is embedded in the file. Recommended for streaming data and schema evolution scenarios. |
| Excel (.xlsx / .xls) | ✅ | ❌ | Reads a specific sheet by name or index. Handles merged cells and automatic type casting. Write to Excel is not supported — use CSV as export format. |
| XML | ✅ | ❌ | Flattens hierarchical XML into tabular rows using a user-defined Row Tag. Supports XML namespaces and attributes. Write to XML is not supported. |
| Delta | ✅ | ✅ | Delta Lake table format. Supports ACID transactions, time travel, and schema enforcement. Requires Delta Lake libraries on the Spark cluster. |
| Iceberg | ✅ | ✅ | Apache Iceberg table format. Requires Iceberg catalog configuration (REST, Glue, Hive, Nessie) on the Spark cluster. |
Compression
DataKnits supports reading and writing compressed files for text-based formats (CSV, JSON). Supported codecs: gzip, bzip2, lz4, snappy, deflate. For Parquet and ORC, compression is configured at the Spark session level (default: Snappy).
Compute & Orchestration Backends
Compute connectors hold no data of their own — they define where generated PySpark actually runs. Set one as the parent compute connector on a data connection to control execution.