Senior Data Engineer

Job Description

A rapidly growing Wagering Tech company in Australia is actively looking for a Senior Data Engineer with solid experience in Databricks, Power BI & Python to be a part of the growing talent pool in Colombo, Sri Lanka.

The engineer in this role is responsible for designing, building, and maintaining data processing systems using Databricks and Apache Spark with a focus on the Python programming language, as well as integrating these systems with Power BI for data visualization and reporting.

Responsibilities
• Collaborating with data scientists and analysts to identify and solve complex data processing challenges, and presenting the results in Power BI dashboards.
• Becoming the primary go to person for data ingestion process from Azure Event Hubs into Databricks.
• Implementing data ingestion and transformation processes using Databricks, Apache Spark, and Python.
• Designing and building data storage solutions, such as data lakes and data warehouses, and integrating them with Power BI for data visualization and reporting.
• Monitoring, Diagnosing & Resolving functional & non functional issues across Databricks, Power BI & Python code.

Requirements
• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• Strong hands-on experience with Databricks and Apache Spark.
• Experience with cloud platforms such as Azure or AWS, including data storage and processing services like S3, Glue, and EMR.
• Experience in using Databricks Notebooks.
• Proficiency in Python.
• Understanding of how to set up Databricks & Power BI for Prod & Non-Prod environments.
• Understanding of handling large datasets in near real time.
• Excellent communication, collaboration, and problem-solving skills.
• Strong ability to work independently and in a team environment.

Skills that are highly beneficial
• Exposure to Protobuf
• Exposure to .NET
• Ability to troubleshoot issues in the integrated systems

Interested candidates, please send you CV to gihan@mytechjobs.lk