The Sr. Big Data Engineer’s primary responsibilities are to build, integrate data from various resources and support Cogensia’s big data ecosystem. The Sr. Engineer will work closely with data architects to design and implement optimum solutions using best practices. They are responsible to ensure the data ecosystem is built to be highly scalable and responsive through writing complex queries and ensuring optimal performance and availability.
The Big Data Engineer also collaborates with Cogensia’s data scientists and account team to creates ETL, batch, and automated processes to best suite Cogensia and its client’s needs.
They heavily contribute to the teams coding and programming standards, offer suggestions to how better improve them, and ensure junior staff are following the same guidelines. They have a high standard for the quality of code and documentation they themselves produce.
- Work closely with SMEs, make solution recommendations, and implement agreed upon solutions using best practices.
- Select and integrate any Big Data tools and frameworks required to provide requested capabilities.
- Design and implement ETL and automated processes.
- Monitor performance and advise of any necessary improvements and changes.
- Management of EMR clusters, Glue Jobs, Athena Tables, S3 data lakes; with all included services.
- Provide technical support to members of TS and SA team, as well as project support across client engagements.
- Work with geographically dispersed teams, embracing Agile and DevOps strategies for themselves and others while driving adoption to enable greater technology and business value.
- Stays current with relevant technology in order to maintain and/or improve functionality for authored applications.
- Assume other responsibilities as requested/required.
- Acts as a subject matter expert for systems worked on. Ensures Cogensia’s data solutions are using the latest versions and code base.
- Actively listen to and work with end users to gather feedback and input, and make suggestions and solutions based on said feedback.