Data Engineer

Responsibilities

• Architect for RAG: Design and scale pipelines for Retrieval-Augmented Generation (RAG), transforming large volumes of unstructured IT logs and documentation into optimized vector embeddings.
• Scale vector infrastructure: Oversee the health and performance of vector databases (Pinecone, Milvus, Weaviate), ensuring sub-second retrieval speeds for agentic reasoning loops.
• Engineer semantic layers: Build knowledge graphs and semantic layers beyond simple ETL to provide agents with the necessary context for navigating complex infrastructure puzzles.
• Automate data excellence: Build automated guardrails to detect noise, bias, or PII before it reaches the model.
• Bridge raw, messy data sources and deep technical AI work, identifying and resolving quality issues at the source.
• Progress to production: Build, deploy, and maintain CI/CD pipelines for data infrastructure, ensuring that the context window re...