Acting as the software engineer of the group to perform design and implementation tasks for our various platforms. Main accountabilities are
- Designing and implementing ingest, processing and curation solution using Big data technologies
- Identify, resolve, and/or escalate all issues that threaten the successful delivery of assigned initiatives
- Utilizing open source tool sets related to the Hadoop ecosystem.
- Create high quality API's for data accessibility on the Hadoop platform that ultimately integrate with the broader environment.
- Using big data analytics techniques provide data and analytics output as required to the broader community including business partners in a manner they can understand to ensure communication and buy in.
- Producing high quality code with excellent test coverage
- Code reviews and Efficient and effective code maintenance
- Analysing product requirements and breaking down into user stories and tasks
- Estimating and planning to assist with the development of the product roadmap
- Mentoring junior data engineers
- Developing build, test and deployment automation
- Research and development of potential new technologies
- Defining processes and standards within the team and contributing to the continuous improvement of the product development process
- Producing a high standard of technical documentation
- Working with customers to understand requirements and technical issues
- Ability to provide technical expertise for new and ongoing initiatives to work with project stakeholder to come up most feasible technical solution.
Technology skills
- Proficiency in Scala and Java is essential.
- Must have experience working with real-time ingestion, processing, and curation of data at big data scale.
- Big Data developer experience with hortonworks or Cloudera would be beneficial.
- Fundamental knowledge of Hadoop components such as Kafka, Spark, Hive, HBase etc. is mandatory.
- Previous experience developing data pipelines using Spark.
- Past experience with NoSQL data stores such as ElasticSearch, MongoDB, Couchbase etc would be beneficial.
- Experience on data analytics using SparkML or other ML tools such as R is a good to have.
Qualifications/experiences
- Demonstrated technology subject matter expertise.
- Familiarity with technology hands on configuration.
- Able to communicate effectively with stakeholders.
- Able to comprehend and deliver to a schedule.
- Ability to work under pressure to deliver to near-term deadlines.
- Ability to work in a matrix organisation, taking direction from both line and project management.
- Self-starter, able to get up to speed quickly and work independently.
- Solid experience working in software development
- Experience working on Big data technologies
