RemoteWLBRemote only
Back to all jobs
Saviynt logo
Remote role · principa

Principal Software Engineer, AI Platform Engineering

Saviynt

hybrid,Milpitas, CaliforniaSoftware DevelopmentFull-time
Salary$274k - $304k
Before you apply

Role details

The practical picture
Employment
Full-time
Role signal brief

The role, at a glance

Extracted from the source posting · 95% confidence

Saviynt is seeking a Principal Software Engineer to architect and manage the AI data platform, focusing on multi-tenant data lakes, batch and streaming pipelines, and vector databases on Google Cloud. The role requires deep expertise in Spark, Dataflow, and orchestrators like Flyte to ensure secure, PII-free data flows for ML training.

Compensation
USD 274,000–304,000 / year

What you need

  • Data lake architecture and operations (Iceberg, Delta Lake, GCS)
  • Apache Spark (PySpark/Scala) and Dataproc
  • Apache Beam and Google Cloud Dataflow
  • Schema Registry (Protobuf/Avro) versioning and migrations
  • Workflow Orchestration (Flyte, Kubeflow, Airflow, or Prefect)
  • Multi-tenant data isolation and governance (PII/Compliance)

What you will own

  • Design and manage the AI Data Lake on GCS, including tier separation, encryption, and lifecycle rules.
  • Develop and maintain batch pipelines using Spark on Dataproc for feature backfills and Iceberg compaction.
  • Build streaming ingestion pipelines using Apache Beam on Dataflow with CDC support and PII filtering.
  • Implement and maintain a schema registry for versioning and compatibility management of Avro and Protobuf schemas.
  • Standardize orchestration workflows using Flyte, defining task authoring standards, isolation, and retry policies.
  • Enforce multi-tenancy through strict per-tenant storage isolation, quotas, and contamination validation.
Core skills
Data EngineeringApache SparkGCP (GCS, Dataproc, Dataflow)Kubernetes
Internal freshness window ends Oct 21, 2026
View source evidence
  • must_have “Data lake ownership (essential): you have designed and operated a production data lake end-to-end”
  • must_have “Deep Spark (PySpark / Scala): executor tuning, shuffle diagnosis, Iceberg table maintenance”
  • must_have “Hands-on Beam / Dataflow: windowing, exactly-once, side inputs, autoscaling”
  • must_have “Schema registry experience: Protobuf / Avro compatibility rules, breaking-change migrations in production”
  • must_have “Orchestration at scale: Flyte, Kubeflow Pipelines, Airflow, or Prefect — operated in production”
  • must_have “Feature store operations: Feast or Tecton, point-in-time joins, online/offline consistency”
  • must_have “Vector databases: Pgvector or Qdrant in production — index tuning, ANN search, embedding upsert pipelines”
  • must_have “RAG data fundamentals: chunking strategies, embedding model selection, retrieval quality evaluation, and context freshness management”
  • must_have “Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience or equivalent military experience”
  • salary “274000-304000 USD/yr”
  • location “hybrid,Milpitas, California”
data engineeringapache sparkgcp (gcs, dataproc, dataflow)kubernetes
Role brief

ABOUT SAVIYNT

Saviynt is a leader in identity security, delivering an AI-powered platform that governs and secures access to applications, data, and business processes for global enterprises and government institutions. Built for the AI era, Saviynt helps organizations move faster — securely and compliantly.

 

ABOUT THE ROLE

You set the architectural direction for how training data flows, evolves, and is governed across the AI Platform. You define the standards ML engineers and scientists build on, and ensure every training signal is tenant-isolated, PII-free, and traceable from source to model.

 

WHAT YOU'LL OWN

  • AI Data Lake on GCS: bucket layout, raw → silver → gold tier separation, CMEK encryption, lifecycle rules

  • Batch pipelines: Spark on Dataproc for TB-scale feature backfills, Iceberg compaction, and daily S3→GCS incremental sync

  • Streaming pipelines: Apache Beam on Dataflow for sub-5-min CDC ingestion with exactly-once semantics and PII assertion gates

  • Schema registry: Avro / Protobuf schema versioning, compatibility modes, and migration playbooks for safe schema evolution

  • Orchestration: Flyte as primary DAG layer — task authoring standards, domain isolation, retry policies, DataCatalog memoization; evaluate Kubeflow Pipelines where relevant

  • Multi-tenancy: strict per-tenant GCS prefix isolation, quota policies, and cross-tenant contamination validation

  • Data Anonymizer and Data Labeler microservices: strip PII and attach ML labels before signals leave each customer environment

  • Feature store: Feast offline (GCS Parquet) and online (Redis) with point-in-time correctness and < 0.1% consistency SLA

  • Vector database: operate Pgvector (Cloud SQL) for POC and Qdrant on GKE for production-scale embedding storage; design index strategies (IVFFlat, HNSW) and manage ANN query latency SLAs

  • RAG data pipeline: build embedding generation pipelines that chunk, encode, and upsert document embeddings into the vector store; own the data refresh cadence and staleness SLAs for retrieval context

  • Service APIs: expose data platform services (feature serving, embedding upsert, schema validation) over HTTPS with mTLS and gRPC where low-latency streaming is required

  • Synthetic data pipelines for dev/staging where real customer data is not permitted

  • Data quality gates: Great Expectations / dbt checks as Flyte tasks, blocking on schema and PII-absence failures

 

YOU'LL THRIVE HERE IF YOU HAVE

  • 1+ years of experience as a Principal SWE at a SaaS company

  • Demonstrated principal impact: platform standards you defined adopted org-wide, or major cross-team pipeline/schema migrations you led

  • Data lake ownership (essential): you have designed and operated a production data lake end-to-end — storage layout, partitioning strategy, tiered retention (hot/warm/cold), table format (Iceberg or Delta Lake), compaction, and access control; not just consumed one

  • Deep Spark (PySpark / Scala): executor tuning, shuffle diagnosis, Iceberg table maintenance

  • Hands-on Beam / Dataflow: windowing, exactly-once, side inputs, autoscaling

  • Schema registry experience: Protobuf / Avro compatibility rules, breaking-change migrations in production

  • Orchestration at scale: Flyte, Kubeflow Pipelines, Airflow, or Prefect — operated in production, ideally benchmarked two

  • Multi-tenant data architecture: per-tenant isolation as a hard requirement, not a post-hoc concern

  • Feature store operations: Feast or Tecton, point-in-time joins, online/offline consistency

  • Vector databases: Pgvector or Qdrant in production — index tuning, ANN search, embedding upsert pipelines

  • RAG data fundamentals: chunking strategies, embedding model selection, retrieval quality evaluation, and context freshness management

  • API transport: gRPC and HTTPS/mTLS for service-to-service communication; comfortable defining proto contracts and managing certificate lifecycle

  • Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience or equivalent military experience

 

NICE TO HAVE

  • Differential privacy or k-anonymity for ML training datasets

  • Open source contributions: Feast, Great Expectations, Apache Beam, or dbt

  • Familiarity with IAM / access governance data: entitlements, provisioning events, access graphs

  • Iceberg or Delta Lake at petabyte scale

 

WHY JOIN SAVIYNT

  • Work on a large-scale, Kubernetes-based SaaS platform

  • Solve challenging cloud and reliability problems at scale

  • Collaborate with strong engineers in a reliability-focused culture

  • Competitive compensation, benefits, and growth opportunities

 

SECURITY & COMPLIANCE

This role requires adherence to Saviynt's information security and privacy policies, including annual security training.