Developer Programmer / Developer

12 years experience working in large, complex Data Warehouse, Engineering, Data Lakehouse environments.

Work Location: The Foundry, 1 Locomotive St, Eveleigh NSW 2015

• Primary or Mandatory skills:

Cloud based Data Warehouse and Lakehouse platforms, Big Data Tech Stack, Hadoop, Spark, SparkSQL, Pyspark, Python, DevOps, DevSecOps, Git, SQL.

• Good to have skills:

Power BI, BMC Control-M, Confluence, Jira, Jenkins, DB Knowledge of Teradata, Oracle, SQL-Server.

• Role description/Main duties:

Objective is to ingest data into Cloud/Hadoop Data Lake from various sources and transform the data as per business requirements. As part of data ingestion, will come across different patterns and would need to build code accordingly. This will involve code development based on the ongoing requirements and perform testing activities to meet the requirement. Will be involved in code deployment into Production, Scheduling of jobs and perform Application Maintenance. Also, will be involved in the Preparation of Technical and functional Specification documents based on requirements. Perform data profiling, analysis, and create ad-hoc BI reports as needed.

• Responsibilities:

1. Analyzing the requirements provided by clients and preparing low level/high level documentation to Confluence pages.

2. Responsible for preparing the technical specifications for the new functionalities as required to be implemented for the Impacted Programs.

3. Mentoring and providing guidance to junior team members, helping them to develop their skills and expertise in software engineering.

4. Perform POC development for technical solution feasibility using prototyping tools where necessary.

5. Perform coding and unit testing based on approved design specifications and provide support for business and functional acceptance testing which involves ensuring technical specifications meet business requirements and drive an understanding and adherence to enterprise architecture standards.

6. Implement optimization techniques, performance enhancement of queries for improving the data load performances in various applications.

7. Write unit test cases using TDD and perform unit testing, prepare support documentation during the UAT phase, and give transition to baseline teams.

8. Designing and scheduling jobs using Control-M, deliverables and assisting technical reviews during all phases of SDLC, code migration from development to test and production environments.

9. Continually improve and automate processes to drive increased flow through our systems to enable Agile Delivery.

10. Accountable for addressing customer needs and maintaining the flow of working solutions based on customer demands.

11. Improve and actively participate in CI/CD and DevOps/DevSecOps practices.

12. Involve in migrating Development code into QA and Production.

13. Support code deployment activities and provide warranty support and fix any issues found during post-deployment phase.

• Essential Skills required to perform the above job role:

1. Expertise in data ingestion and integration patterns – batch or data streaming, parallel or iterative ingestion of multiple data sources.

2. Designs data staging areas to land data in Data Lake from multiple source systems.

3. Understanding the Business requirements and capable of presenting an end-to-end data flow.

4. Design and implement the data ingestion pipelines from multiple sources into Azure data lake (Azure blob, ADLS) using ADF, Azure Databricks.

5. Design and development of Spark applications using Spark SQL/PySpark in Databricks for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.

6. Expertise with Hadoop echo system (Hdfs, Hive, Sqoop, Spark SQL, PySpark), Teradata and Oracle, SQL-Server.

7. Handle process scheduling and automation using tools like Control-M, Azure DevOps, Jenkins.

8. Good experience in building, designing, and maintaining cloud-based applications with Azure.

9. Effective leadership and cross-functional leadership and communication skills.

10. Implement optimization techniques, performance enhancement of queries for enhancing the data load performance in various applications.

11. Expertise in operating Linux environment with good command over scripting language such as Shell, Python, etc.

12. Source Control version tools such as Bit bucket, Git and CI/CD deployments using DevOps, DevSecOps practises.

13. Design, develop and maintain BI dashboards using Microsoft Power BI from multiple data sources to facilitate data driven decision making.

• Special Skills required to perform the job role:

1. Good knowledge of ingestion patterns and stream ingestions.

2. Experience in building complex data transformation rules.

3. Working knowledge of light weight data transformation, curation, conformed data provision using Big Data, Azure databricks, Python, Spark, Teradata, and others.

4. Experience in Python and SQL scripting to create Spark jobs, query optimization.

5. Optimize Power BI dashboards with a focus on usability, performance, flexibility, testability, and standardization.

• Looking for candidates available to work:
  • Monday: Morning, Afternoon, Evening
  • Tuesday: Morning, Afternoon, Evening
  • Wednesday: Morning, Afternoon, Evening
  • Thursday: Morning, Afternoon, Evening
  • Friday: Morning, Afternoon, Evening
• More than 4 years of relevant work experience required for this role
• Working rights required for this role
• Expected salary: $135,000 - $145,000 per year