Senior Data Engineer, Ingestion Framework - Data & AI Platform

Role Summary As a Senior Data Engineer in the Data & AI Platform team, you will play a pivotal role in enabling our big data platform to operate seamlessly at a petabyte scale. The successful candidate will be an architect and custodian of a custom-built, Go‑powered vehicle data processing framework, ensuring its smooth integration with our Databricks platform.

Responsibilities

Framework Development & Maintenance: Design, develop, and enhance a cutting‑edge, petabyte‑scale vehicle data processing framework using Go. This framework is the backbone of our data operations; therefore, the ability to write clean, efficient, and maintainable code is essential.

Databricks Integration: Collaborate closely with the big data platform team to ensure seamless integration of the framework with our Databricks environment. This involves understanding the intricacies of both systems and optimizing their interaction for maximum efficiency.

CI/CD & Kubernetes Manageme...