Senior Data Engineer

This position is based in the Austroads Melbourne or Sydney offices.

The Senior Data Engineer will assist in the design, development, enhancement, and maintenance of complex data pipelines products that manage business critical operations, and large-scale analytics pipelines. Reporting to the Head of TCA Application Delivery, the Senior Data Engineer will work on Big Data systems design focusing on data architecture in our Databricks ecosystem.

Major responsibilities/accountabilities

The role is responsible for:

• Designing and development of data insights processes (SQL/Azure Cloud based), the data model and the business operations
• Providing technical insights towards functional and non-functional requirements for the new and existing features within the Databricks ecosystem
• Scoping and executing required solutions together with a multidisciplinary team that includes Data Analysts, Data Administrators, GIS Analysts and Business SMEs to understand platform capabilities and how to best improve and expand those capabilities to support client requirements
• Designing, developing, enhancing, and maintaining complex data pipeline products which manage business-critical operations and large-scale analytics applications
• Supporting analytics, data science and/or engineering teams and understanding their unique needs and challenges
• Embrace the processes, methodologies, standards, and technology choices
• Contributing to continuous improvement, identifying ways to enhance value for our members and the public
• Maintaining and adherence to Quality Systems processes and procedures associated with the role
• Other duties as directed.

Qualifications, knowledge, and experience

The Senior Data Engineer will possess:

Required

• Demonstrated experience in Big Data solutions in a cloud ecosystem, including Data/Delta Lakes
• Demonstrated experience with Big Data pipelines or DAG Tools such as Azure Data Factory, Spark (especially Databricks, Spark and Delta Lake)
• Hands-on Python/Pyspark/SparkSQL experience; Scala experience is a plus
• Experience with source control (GitHub Enterprise)
• SQL syntax experience, specifically to write complex, highly optimised queries across large volumes of data
• Demonstrated experience working in an Agile Scrum/Kanban environment as well as delivery-based test/release approach.

Desirable

• Data modelling/Data profiling capabilities with Kimball/star schema methodology
• Experience with Kafka, or other live data streaming technology
• Experience with database deployment pipelines
• Experience with one or more unit testing or data quality frameworks
• Experience in working with Telematics Data and Road Maps
• Experience with conceptual, logical and/or physical database designs is a plus.

Personal qualities

The incumbent will need to present a professional image and build a strong relationships across the enterprise. Specifically, the role requires:

• Self-management and personal leadership
• Sound judgement and problem-solving ability
• Strong networking and teamworking skills
• Strong attention to detail
• Commitment to ongoing professional development and learning
• Professional ethics and integrity
• Comfort working in an environment which is constantly evolving

Apply for the Senior Data Engineer position.

View a copy of the position description