Job details

Data Engineer

Job reference: 003695

Closing date: 03/09/2021

Salary: Competitive

Department: IT

Hours per week: 36.17 hours per week

Data Engineer – Data Platform Squad

Job Description

We are looking for a Data Engineer with a minimum of 3 years of hands-on experience to bring in-depth technical knowledge of building and supporting data pipelines using cloud-native services and demonstrate strong interpersonal communication skills. Experience of using serverless cloud services to build robust pipelines following both batch and near real-time ingestion patterns for a large-scale Data Lake or equivalent is preferred.

You will have a large amount of autonomy to do what is needed to succeed, a clear and meaningful purpose, and backing to demonstrate mastery of the trade. You will also have the support and freedom to employ industry-leading engineering practices, with a clear focus on customer needs.

Your Responsibilities

As a Data Engineer you are integral to the success of a data driven business. You will work in a cross-functional Agile team, orchestrating automated data pipelines using big data tools and infrastructure and seeing changes through into production. You will be working within a cloud-based environment, and therefore experience in this area is required

By delivering data that is reliable, trustworthy and highly available and also focusing on security and performance during every stage of development, you will be confident that with each release the codebase will perform up to high quality standards. You will not be afraid to challenge ideas and opinions to avoid pitfalls and inefficient solutions.

The ideal candidate will

  • Be technically adept and have a deep hands-on knowledge of data engineering patterns, practices and architectural principles.
  • Be highly motivated, driven and adaptable.
  • Be passionate about learning.
  • Have experience providing mentorship or knowledge-transfer to team members.
  • Leverage the right tools for the right job to deliver modern data solutions that are testable and maintainable.
  • Have demonstrable experience in maintaining strong technical standards in a fast-paced environment.
  • Be familiar with Pair Programming and an interest in helping to grow the skills of your fellow engineers.

and can

  • Help lead the implementation and continuous improvements of a data lake architecture, optimised for analysis and machine learning.
  • Apply extensive AWS Knowledge in S3, IAM, Cloudwatch, EC2, Glue, Athena, Lambda and Kinesis.
  • Deliver data through various data ingestion options such as batch, stream and messaging from files, databases and APIs.
  • Get stuck into scripting for both data and file processing, e.g. SQL, Python, Shell, Perl.
  • Work with various file and data formats such as JSON, XML, Parquet, Avro, CSV, Fixed Width.
  • Contribute effortlessly within an Agile, DevOps process methodology to CI/CD automation, Version Control (Git) & Test-Driven Development.
  • Use data orchestration platforms, preferably Apache Airflow.