Senior Data Engineer (Kafka Streaming, Spark, Iceberg on Kubernetes) (Remote)

  • Remote
  • Anywhere

The Legends Agency

Senior Data Engineer (Kafka Streaming, Spark, Iceberg on Kubernetes)
Build and scale a next-generation real-time data platform with cutting-edge open-source technologies.
100% Remote | R100 000 – R110 000 per month

About Our Client

Our client is a rapidly growing technology-driven organization building high-performance data platforms to enable advanced analytics, AI, and business intelligence. The team operates at the forefront of real-time data processing and distributed systems, leveraging modern cloud-native infrastructure. They foster a culture of technical excellence, continuous learning, and collaboration across multidisciplinary engineering teams.

The Role: Senior Data Engineer

As a Senior Data Engineer, you will design, build, and optimize next-generation data pipelines and platforms. Youll lead the architecture and implementation of scalable, real-time data solutions using Kafka, Spark, and Apache Iceberg deployed on Kubernetes. This is a hands-on, high-impact role within a forward-thinking data engineering team focused on performance, scalability, and innovation.

Key Responsibilities

  • 5+ years of professional experience in data engineering or software engineering

  • Design and implement scalable, highly available real-time data pipelines and architectures

  • Build robust ETL and streaming pipelines using Apache Spark (Scala/Python) and Kafka Connect/Streams

  • Develop and manage data lakes using Apache Iceberg with schema evolution and time travel capabilities

  • Deploy and manage distributed data processing services on Kubernetes using containerization best practices

  • Optimize performance and resource usage across Spark jobs, streaming apps, and Iceberg tables

  • Define and uphold engineering best practices including testing, code standards, and CI/CD workflows

  • Mentor junior engineers and contribute to building a high-performing data engineering team

About You

  • 5+ years of experience in data engineering or related software engineering roles

  • Advanced proficiency with Apache Spark (batch and streaming)

  • In-depth experience with Apache Kafka (Connect, Streams, or ksqlDB)

  • Hands-on experience with Apache Iceberg, including table evolution and performance tuning

  • Skilled in Python (PySpark) or Scala

  • Experience deploying and managing distributed systems on Kubernetes (Spark Operator is a plus)

  • Solid understanding of data modeling and data warehousing concepts

  • Advantageous: Experience with AWS, Azure, or GCP; familiarity with Flink or Trino

  • Preferred: Bachelors or Masters degree in Computer Science, Engineering, or related field

Originally posted on Himalayas

To apply for this job please visit himalayas.app.