Description:
We are seeking a Senior Software Engineer specializing in data pipelining and modeling. You will play a crucial role in designing, developing, and maintaining our data infrastructure. You will work closely with cross-functional teams to build robust, scalable, and efficient data pipelines, ensuring high data quality and reliability. Your expertise in ETL processes, Airflow, Streaming, Warehousing, and database technologies will drive our data initiatives forward.
What you’ll do:
- Design and implement data pipelines for ingesting, processing, and transforming large volumes of data.
- Develop and maintain data models to support analytical and reporting needs.
- Optimize data pipelines for performance, scalability, and reliability.
- Implement real-time data streaming solutions using streaming technologies.
- Create and manage ETL processes to extract, transform, and load data from various sources into data stores.
- Monitor and troubleshoot data pipeline and ETL issues.
- Evaluate, select, and implement columnar and store databases that best-fit project requirements.
- Understand the tradeoffs between different database technologies and make informed decisions.
- Perform database optimization and tuning for efficient data retrieval and storage.
- Collaborate closely with data scientists, analysts, and other engineering teams to understand data requirements and deliver solutions.
- Document data pipelines, models, and ETL processes for knowledge sharing and troubleshooting.
- Promote and enforce best practices in data engineering and data governance.
- Participate in an on-call rotation schedule to provide timely response and support for engineering-related issues outside of regular business hours, ensuring the continuous operation of critical systems and infrastructure.
Who you are:
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
- Minimum of 5+ years of experience in software engineering with a focus on data engineering.
- Strong proficiency in data pipelining and modeling, including experience with tools like Apache Airflow.
- In-depth knowledge of data streaming technologies, especially Apache Kafka.
- Expertise in designing and implementing ETL processes.
- Proficiency in SQL and NoSQL databases, understanding the tradeoffs between different types.
- Experience with columnar and store databases.
- Strong problem-solving and debugging skills.
- Excellent communication and teamwork abilities.
- Familiarity with cloud platforms like AWS, GCP, or Azure is a plus.