The Sr. Big Data Engineer’s primary responsibilities are to build,
integrate data from various resources and support Cogensia’s big data
ecosystem. The Sr. Engineer will work closely with data architects to
design and implement optimum solutions using best practices. They are
responsible to ensure the data ecosystem is built to be highly
scalable and responsive through writing complex queries and ensuring
optimal performance and availability.
The Big Data Engineer also collaborates with Cogensia’s data
scientists and account team to creates ETL, batch, and automated
processes to best suite Cogensia and its client’s needs.
They heavily contribute to the teams coding and programming
standards, offer suggestions to how better improve them, and ensure
junior staff are following the same guidelines. They have a high
standard for the quality of code and documentation they themselves produce.
- Work closely with SMEs, make solution recommendations, and
implement agreed upon solutions using best practices.
- Select and integrate any Big Data tools and frameworks required to
provide requested capabilities.
- Design and implement ETL and automated processes.
- Monitor performance and advise of any necessary improvements and changes.
- Management of EMR clusters, Glue Jobs, Athena Tables, S3 data
lakes; with all included services.
- Provide technical support to members of TS and SA team, as well as
project support across client engagements.
- Work with geographically dispersed teams, embracing Agile and
DevOps strategies for themselves and others while driving adoption
to enable greater technology and business value.
- Stays current with relevant technology in order to maintain and/or
improve functionality for authored applications.
- Assume other responsibilities as requested/required.
- Acts as a subject matter expert for systems worked on. Ensures
Cogensia’s data solutions are using the latest versions and code base.
- Actively listen to and work with end users to gather feedback and
input, and make suggestions and solutions based on said feedback.