AI/ML Data Engineer (Databricks)
QuidelOrtho is a leading in vitro diagnostics company formed from the merger of Quidel Corporation and Ortho Clinical Diagnostics. They are seeking an AI/ML Data Engineer to design, build, and optimize data pipelines and infrastructure using Databricks for AI and machine learning initiatives, collaborating with business stakeholders to translate requirements into technical solutions.
Responsibilities
- Work directly with business stakeholders to identify and define AI/ML use cases, translating business needs into technical requirements
- Design, develop, and optimize scalable data pipelines in Databricks for AI/ML applications, ensuring efficient data ingestion, transformation, and storage
- Build and manage Apache Spark-based data processing jobs in Databricks, ensuring performance optimization and resource efficiency
- Implement ETL/ELT processes and orchestrate workflows using Azure Data Factory, integrating various data sources such as Azure Data Lake, Blob Storage, and Microsoft Fabric
- Collaborate with Data Engineering teams to meet data infrastructure needs for model training, tuning, and deployment within Databricks and Azure Machine Learning
- Monitor, troubleshoot, and resolve issues within Databricks workflows, ensuring smooth operation and minimal downtime
- Implement best practices for data security, governance, and compliance within Databricks and Azure environments
- Automate data and machine learning workflows using CI/CD pipelines through Azure DevOps
- Maintain documentation of workflows, processes, and best practices to ensure knowledge sharing across teams
- Perform other work-related duties as assigned
Skills
- Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent experience)
- 1-3 years of experience in data engineering, with a strong focus on Databricks and AI/ML applications
- Proven experience working directly with business stakeholders to identify and implement AI/ML use cases
- Expertise in Apache Spark and hands-on experience with Databricks for building and optimizing data pipelines
- Strong programming skills in Python and Scala for data engineering and machine learning workflows in Databricks
- Experience with Azure Data Factory, Azure Data Lake, Azure Blob Storage, and Azure Synapse Analytics
- Proficiency with Databricks Delta Lake for data reliability and performance optimization
- Familiarity with MLflow and Databricks Runtime for Machine Learning for model management and deployment
- Knowledge of Azure DevOps for implementing CI/CD pipelines in Databricks-based projects
- Strong understanding of data governance, security practices, and compliance requirements in cloud environments
- Familiarity with emerging Databricks features such as Delta Live Tables and Unity Catalog
- Ability to travel up to 5-10%
- This position is not currently eligible for visa sponsorship
- Experience with real-time data processing using Apache Kafka or Azure Event Hubs
- Master's degree in Computer Science or related technical fields
Benefits
- Bonus eligible
- Medical, dental, vision, life, and disability insurance
- 401(k) plan
- Employee assistance program
- Employee Stock Purchase Plan
- Paid time off (including sick time)
- Paid Holidays
Company Overview