RemoteWLBRemote only
Back to all jobs
Rackspace logo
Remote role · azure-se

Azure - Senior Data Engineer - Synapse

Rackspace

hybrid,Vietnam - RemoteData
Role brief

Description

Design, develop, and maintain real-time data streaming pipelines using Spark Streaming or Azure Functions. We are seeking a highly skilled Senior Data Engineer or Data Architect with extensive experience in Synapse and Azure data services. The ideal candidate will have a deep understanding of Spark and Lakehouse architectures, as well as a proven track record of implementing and managing data solutions using Synapse services and other Azure tools. 

Design, develop, and maintain real-time data streaming pipelines using Spark Streaming or Azure Functions. We are seeking a highly skilled Senior Data Engineer or Data Architect with extensive experience in Synapse and Azure data services. The ideal candidate will have a deep understanding of Spark and Lakehouse architectures, as well as a proven track record of implementing and managing data solutions using Synapse services and other Azure tools. 

Key Responsibilities

  • Design, develop, and maintain data architectures using Spark and Lakehouse methodologies. 
  • Utilize Synapse services including Spark pools, dedicated SQL pools, and pipelines to build robust data solutions. 
  • Leverage other Azure data services to enhance and optimize data workflows. 
  • Implement and manage CI/CD pipelines to ensure smooth deployment and integration processes. 

    • Qualifications

  • Strong expertise in Spark and Lakehouse architectures. 
  • Hands-on experience with Synapse services: Spark pools, dedicated SQL pools, and pipelines. 
  • Proficiency with additional Azure data services. 
  • Solid experience with CI/CD processes and tools. 
  • Excellent problem-solving skills and the ability to work in a fast-paced environment. 
  • Join our team and contribute to cutting-edge data solutions in a dynamic and innovative setting. 
  • Load, merge, and process machine logs from Kafka, ensuring efficient data flow and transformation. 
  • Integrate processed data into Redis cache and send it to the data lake for long-term storage and analysis. 
  • Implement and optimize data processing solutions using Python. 
  • Apply software engineering best practices, including code reviews, version control, and continuous integration/deployment (CI/CD). 
  • Collaborate with cross-functional teams to understand requirements and deliver high-quality data solutions. 

    • Requirements: 
    ·         Proven experience in real-time data streaming using Spark Streaming or Azure Functions.
    ·         Strong proficiency in Python for data processing and automation tasks.
    ·         Hands-on experience with Kafka for data ingestion and message queuing.
    ·         Familiarity with Redis for caching and fast data retrieval.
    ·         Knowledge of data lake architectures and best practices for data storage and retrieval.
    ·         Solid understanding of software engineering principles, including design patterns, testing, and documentation.
    ·         Strong problem-solving skills and ability to work in a fast-paced, collaborative environment.
    Preferred Qualifications:
    ·         Experience with cloud platforms such as Azure 
    ·         Knowledge of other programming languages and frameworks related to data engineering. 
    ·         Familiarity with data modeling, ETL processes, and data lake concepts.