Description

The ideal candidate is an experienced data pipeline builder who enjoys optimizing data systems and building them from the ground up. The Data Engineer will support software developers, database architects, data analysts and data scientists on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects.

Client Details

Our client is building an IT competency centre in Lisbon and requires technically qualified, adaptable and ambitious IT professionals.

Description

• Create and maintain optimal data pipeline architecture
• Assemble large, complex data sets that meet functional / non-functional business requirements
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS 'big data' technologies
• Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs
• Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader

Profile

• 3+ years extensive experience of building big data pipelines with Apache Spark - real-time and batch processing
• 3+ years experience of interacting with big data and cloud data stores, both SQL and noSQL such as Hive, Presto, Apache Kudu, Big Query, AWS Athena or related big data store systems
• 3+ years' experience of Python
• Experience building large scale distributed systems or applications
• Experience with Kafka, Kinesis, Storm, Spark and other data pipeline technologies
• Strong knowledge of Design Patterns
• Strong data modeling skills (NoSQL & RDBMS)
• Experience with Linux, good knowledge of Linux administration
• Experience with Unix Shell Scripting - Experience with cloud technologies such as Amazon Web Services (AWS), Microsoft Azure or Google Cloud Platform
• Experience HAProxy, NginX, Netty, Docker
• Exposure to modern execution frameworks like Mesos or Kubernetes as well as management tools like Terraform
• Experience with Continuous Integration, QA, Stress Testing
• Experience with configuration management tools like Puppet, Chef, Ansible or Saltstack
• Strong knowledge of Agile development methodology

Job Offer

Join a leading company.

Candidate-se através do website