Senior Data Engineer

Position Overview: As a Senior Data Engineer, you will play a crucial role in designing, implementing, and maintaining robust data solutions in alignment with the technical requirements defined by our System Analysts that drive operational excellence and support our clients' digital transformation initiatives. This position focuses on leveraging cutting-edge AWS components and data engineering practices to ensure efficient data extraction, transformation, loading, and analysis. A key component of our approach will be the Data Mesh concept, which will guide our development efforts. You will act as a key technical resource, collaborating with internal teams and clients to deliver innovative data solutions that meet business needs and enhance decision-making processes. Key Responsibilities: - Design, implement, and maintain robust ETL pipelines on AWS, ensuring efficient incremental data loading and detailed logging. - Write, optimise, and manage complex SQL queries for data extraction, transformation, and analysis. - Leverage Databricks and Python for data integration, transformation, and analytics workflows. - Strong knowledge of Databricks, Delta Lake and Apache Spark. - Monitor and optimise data pipeline performance, identifying and implementing improvements to enhance efficiency and reliability. - Ensure all data engineering activities comply with company policies, industry regulations, and data security standards. Qualifications: - Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field. - Minimum of 5 years of experience in data engineering roles with a proven track record of designing and implementing data solutions using AWS technologies. - Proficiency in SQL and Python or Scala programming and data management and processing. - Strong knowledge of Apache Spark, and Delta Lake. - Hands-on experience with AWS as hyper-scale cloud platform. - Strong knowledge of data modelling, API integration, and ETL processes. - Excellent problem-solving and analytical skills and a quality mindset. - Excellent communication skills in English (fluency in written and spoken English). - Willingness to travel as needed for client engagements or other professional duties. - Nice to have: An understanding of the Data Mesh concept, Understanding of Azure Data Factory and Azure Services. Working Conditions: This is a remote position based in Portugal. Standard office hours apply; however, flexibility may be required to meet client expectations or service delivery commitments.