About the role:
As an engineer on this team, you will be responsible for building and supporting a petabyte-scale data platform in the cloud and providing powerful foundations for Cruise’s ML Data Platform tools, frameworks, and services. Responsibilities include ensuring scalable, transparent, and reliable data ingestion and management; development of fast, robust, and spike-resistant data consumption, data mining, and processing tools for the entire company; and development of orchestration for large-scale post-processing, and computational pipelines.
What you’ll be doing:
Contribute to the development, optimization and productionization of the next generation data processing platform using Beam and Spark in the cloud.
Build self-serve capabilities to help customers to adopt the next generation data processing platform
Use the latest cloud technologies to own, design, implement, and test scalable distributed data systems in the cloud. Champion engineering excellence by continuously improving systems and processes
Own technical projects from start to finish; contribute to the team’s product roadmap, technical decisions, and tradeoffs; effectively participate in team’s planning, code reviews and design discussions
Consider the effects of projects across teams; work together with partner teams and orgs to achieve cross-organizational goals and satisfy broad requirements
What you must have:
Experience building a data processing system using Beam / Spark and its ecosystems.
Experience optimizing those data processing pipelines for cost efficiency and performance
Experience building serving systems capable of delivering data at high-throughput, low-latency and high QPS in a cost-efficient and spike-resilient manner.
Experience building scalable infrastructure on the cloud with Python or Java/Scala (or similar)
3+ years working with big data
BS, MS or Ph.D. in Computer Science, Electrical Engineering, Mathematics, Physics, or another relevant field; or equivalent real-world experience
Passionate about self-driving technology and its potential impact on the world
Attention to detail and a passion for seeking truth
A track record of efficiently solving complex problems
Startup mentality - openness to dealing with unknown unknowns and wearing many hats
Bonus points!
Demonstrable expertise in a building end-to-end data ingestion, processing and serving systems at petabyte scale
Proficiency in writing SQL queries for analytic purposes
Relevant publications
The salary range for this position is $130,900 - $192,500. Compensation will vary depending on location, job-related knowledge, skills, and experience. You may also be offered a bonus, long-term incentives, and benefits. These ranges are subject to change.