Admin Guide

Platform-wide configuration for Organization Administrators and Super Admins. Covers initial setup, workspace management, connection governance, environment configuration, and compliance audit trails.

Organization Setup

After your first login as Super Administrator, configure the platform-wide settings before adding users or creating workspaces.

System Roles

Initial Organization Configuration Checklist

1. Configure SSO (SAML 2.0 or OIDC) or set local password policy. → SSO guide
2. Set MFA enforcement policy (required / optional / admin-only).
3. Configure IP allowlist for admin access.
4. Create shared connections (databases, cloud buckets) available to all workspaces.
5. Invite the first Workspace Admins and assign them to their workspaces.
6. Set audit log retention policy (default: 90 days; configurable up to 365 days).

Projects & Workspaces

The resource hierarchy is: Organization → Workspace → Project → Folder → Pipeline → Job/Run. Permissions cascade downward unless explicitly overridden at a lower level.

Creating a Workspace

1. Navigate to Admin → Workspaces → New Workspace.
2. Set name, description, and assign a Workspace Administrator.
3. Configure default compute environment (Spark endpoint or Databricks workspace).
4. Optionally bind shared organization connections.

Creating a Project

1. Inside the workspace, go to Projects → New Project.
2. Set name, description, and assign a Project Owner.
3. Configure project-level connections and environment bindings.
4. Add initial team members with appropriate roles (Developer, Analyst, Operator, Approver).

Connection Management

Connections are the credentials and configuration needed to reach a data source or target. They are centrally managed and referenced by pipelines.

Connection Scopes

• Organization-level connections — Shared across all workspaces. Managed by Org Admin.
• Workspace-level connections — Shared across all projects in the workspace. Managed by Workspace Admin.
• Project-level connections — Available only within the project. Managed by Project Owner or Connection Manager role.

Connectivity Methods

Credential Storage

All credentials are encrypted at rest with AES-256 (PostgreSQL pgcrypto extension). Connection managers can create and rotate credentials but can never view plaintext values — all credential fields are write-only via vault after initial entry.

Environments

Environments bind a Spark compute endpoint to an execution context (Development, Staging, Production). Each project can have multiple environments.

• Development — Developers can run pipelines freely. No approval required.
• Staging — Developers can run pipelines. Production-equivalent data; requires same configuration as prod.
• Production — Requires Approver sign-off before a pipeline is promoted. Only Executor/Operator roles can trigger runs.

Each environment stores its own set of runtime variables (e.g., S3 bucket names, database hostnames) which are injected into generated code at execution time — no code changes required to promote from staging to production.

Monitoring & SLA Alerts

DataKnits tracks how long a pipeline has been running and can alert you when a run overstays its expected budget. This is a run-duration SLA — "alert me if this job has been running longer than N minutes" — not a deadline-based "must finish by 6 AM" SLA, and it does not detect a schedule that failed to trigger at all.

There are two tiers, and a run uses whichever applies:

1. Job-level — an explicit threshold set on that pipeline's own alert rule. Always wins when present.
2. Org-wide default — a per-pipeline threshold computed automatically from that pipeline's own history, used only when no job-level rule exists.

Job-Level: Configuring an SLA on a Pipeline

An SLA threshold is one field on a regular alert rule — there is no separate SLA setup screen. Anyone with edit access to a pipeline can configure one from either:

• Pipeline → Alerts tab — scoped to a single pipeline.
• Alerts Center — the global view across all pipelines.

Create a rule, choose event type SLA_VIOLATION or LONG_RUNNING, and a Threshold field appears with a unit selector — enter it in seconds, minutes, or hours, whichever is most natural for that job. Pick a delivery channel (email, Slack, webhook, or PagerDuty), and optionally set a silence window so a stuck job doesn't page you repeatedly.

Org-Wide Default: Percentile-Based SLA

For pipelines nobody has explicitly configured, an admin can turn on an organization-wide default under Settings → SLA Defaults (requires admin/user-management access). Instead of a fixed time value, the default is expressed as a percentile — 75th, 90th, or 95th — of that pipeline's own historical run durations. A pipeline that normally takes 8–12 minutes gets a very different automatic threshold than one that normally takes 3–4 hours, with no manual tuning per pipeline.

• Computed once a day, not live — a background job looks at each pipeline's last 30 successful runs (minimum 5 required before a threshold is trusted) and stores the resulting percentile duration. Nothing is calculated on the fly when a job runs or when a check fires.
• You also set a single default notification channel (type + target) for the org default — that's who gets notified when a pipeline breaches its automatic threshold and has no rule of its own.
• The moment someone adds an explicit job-level rule for a pipeline, that pipeline stops using the automatic default — the job-level rule always takes priority.
• Off by default. Until an admin enables it, behavior is unchanged: only pipelines with an explicit rule get an SLA status.

How Detection Works

Whichever threshold applies (job-level or org-default) is resolved once, the moment a run starts, and stays fixed for that run — it isn't recalculated mid-run even if settings change afterward. A background watchdog polls currently-running pipelines against that stamped threshold and fires the alert (email/Slack/webhook/PagerDuty) the same way any other alert rule fires, with a full audit trail in the Alerts Center (acknowledge, dismiss, or re-notify). A run that completes successfully afterward auto-resolves any open SLA alert for that pipeline.

The poll interval defaults to 60 seconds, but is configurable via the SLA_WATCHDOG_POLL_MS backend variable — see the table below. Lowering it (for example to detect a breach within 5 seconds of the threshold) trades detection latency for more frequent database polling; there is a 1-second floor to prevent a misconfigured value from hammering the database.

SLA Status on Completed Runs

Once a run finishes, its SLA Status (visible in the Monitor view and a run's Execution Detail tab) is set to:

If a pipeline has more than one active SLA rule, the tightest (lowest) threshold is used.

Environment Variables (.env)

DataKnits follows the Twelve-Factor App pattern. Configuration values are read from .env files at startup — never hard-coded.

Backend Variables (Backend/.env)

Frontend Variables (Frontend/.env)

Audit Logging & Compliance

Every significant state change is recorded in the audit log with who, what, when, and from where. Audit triggers are implemented as PostgreSQL triggers — they cannot be bypassed by the application layer.

Audited Events

• User login / logout / failed login attempts.
• User creation, modification, and deactivation.
• Role assignment and revocation.
• Pipeline create, edit, version, publish, delete.
• Connection create, rotate credentials, delete.
• Pipeline execution start, completion, failure, cancellation.
• Admin impersonation sessions.
• Permission override events.

Compliance Standards

DataKnits is architected for compliance with the following standards. See the Security & Trust page for attestation details.

• SOC 2 Type II — audit controls, access management, availability.
• ISO 27001 — information security management.
• GDPR — data minimization; no customer data stored on DataKnits servers.
• HIPAA — supported via Business Associate Agreement (BAA) on enterprise plans.
• PCI-DSS — card data never transits DataKnits (execution-plane separation).