Sr Big Data Engineer

    • Job Tracking ID: 512530-718038
    • Job Location: Schaumburg, IL
    • Job Level: Mid Career (2+ years)
    • Level of Education: BA/BS
    • Job Type: Full-Time/Regular
    • Date Updated: February 20, 2020
    • Years of Experience: 5 - 7 Years
    • Starting Date: February 1, 2020
Invite a friend
facebook LinkedIn Twitter Email


Job Description:

The Sr. Big Data Engineer’s primary responsibilities are to build, integrate data from various resources and support Cogensia’s big data ecosystem. The Sr. Engineer will work closely with data architects to design and implement optimum solutions using best practices. They are responsible to ensure the data ecosystem is built to be highly scalable and responsive through writing complex queries and ensuring optimal performance and availability.

 

The Big Data Engineer also collaborates with Cogensia’s data scientists and account team to creates ETL, batch, and automated processes to best suite Cogensia and its client’s needs.

They heavily contribute to the teams coding and programming standards, offer suggestions to how better improve them, and ensure junior staff are following the same guidelines. They have a high standard for the quality of code and documentation they themselves produce.

 

Responsibilities:

  • Work closely with SMEs, make solution recommendations, and implement agreed upon solutions using best practices.
  • Select and integrate any Big Data tools and frameworks required to provide requested capabilities.
  • Design and implement ETL and automated processes.
  • Monitor performance and advise of any necessary improvements and changes.
  • Management of EMR clusters, Glue Jobs, Athena Tables, S3 data lakes; with all included services.
  • Provide technical support to members of TS and SA team, as well as project support across client engagements.
  • Work with geographically dispersed teams, embracing Agile and DevOps strategies for themselves and others while driving adoption to enable greater technology and business value.
  • Stays current with relevant technology in order to maintain and/or improve functionality for authored applications.
  • Assume other responsibilities as requested/required.
  • Acts as a subject matter expert for systems worked on. Ensures Cogensia’s data solutions are using the latest versions and code base.
  • Actively listen to and work with end users to gather feedback and input, and make suggestions and solutions based on said feedback.

Experience and Skills:

    • (7 years of relevant experience) or (5 years of relevant experience and an advanced degree in Computer Science/IT or related field)
    • Keen understanding of distributed computing principles.
    • Proficiency with Big Data frameworks such as Hadoop, Spark, MapReduce, HDFS.
    • Proven experience ingesting data from multiple data sources such as REST API, SFTP flat files, Streaming data etc.
    • Proven experience with Big Data querying tools such as Athena/Presto, Pig, Hive, and Impala.
    • Proven experience with NoSQL databases, such as HBase, Cassandra, Redshift, DynamoDB.
    • Proven experience with various ETL techniques and frameworks, such as Flume, Glue Jobs, Step Functions.
    • Proven experience with Big Data ML toolkits, such as Mahout, SparkML, or H2O
    • Proven experience with AWS Lambda and leveraging it in various solutions such as Glue, Step Functions, CloudWatch, S3 Events, etc.
    • Strong experience with using Python scripts & libraries.
    • Experience desired with Database Warehousing Design Concepts; Dimensional.
    • Modeling, Star/Snowflake Schemas, ETL/ELT, Data Marts, Analytic Playgrounds, Reporting techniques.
    • Experience working with Agile software development methodologies, namely Scrum.
    • Proven experience with team collaboration, release management, system and performance monitoring.
    • Ability to work well with people from many different disciplines and varying degrees of technical experience.
    • Excellent analytical, problem resolution, organization and time management skills.
    • Ability to handle multiple tasks at a time.