Project overview: Supporting AML migrate their pricing airflow data jobs from hive on emr to spark on kubernetes
Assist with the conversion, refactoring and migration of AML Pricing Airflow DAGs and tasks from Hive running on AWS EMR to Spark running on AWS EKS
This would include performing data validation to ensure the new DAG produces an output equivalent to the original one
Collaborate with David Shu from the DIGIT team as well as Robert Gay towards the overall objective of retiring the AML-hosted Pricing Airflow cluster
As we learn more about the nature of modernizing Hive to Spark as well as organizing our jobs to efficiently perform data validation, a change in direction may be necessary; this will be confirmed by Robert Gay.
Redfin provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination based on race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, and any other characteristic protected by applicable federal, state or local law. If you need accommodation in the application or recruitment process because of a disability or special need, please contact recruitingteam@redfin.com.
Redfin accepts applications on an ongoing basis.