Data Engineer 3 - Rapid Progression

MW Partners is currently seeking a Data Engineer to work for our client who is a global leader in multimedia and creativity software products.

Responsibilities and duties:

Data Asset Inventory and Assessment

• Inventory of existing data assets; tables, dashboards, and pipelines
• Follow requirements to pull required data to measure the current state of these assets, set up usage metrics for internal and external stakeholders.

Table Metadata

• to improve documentation coverage for tables, including table descriptions, column definitions, and data lineage.
• Implement a centralized metadata management system to maintain and access asset documentation.
• Ensure that all existing and new data assets are properly documented according to established standards.
• Pipeline Clean-up and Consolidation
• Consolidate and streamline pipelines by eliminating redundancies and unnecessary elements according to the set of provided rules.
• Clean up and restructure data tables, ensuring consistent naming conventions, data types, and schema definitions.
• Retire or archive obsolete dashboards and workflows.
• Implement monitoring and alerting mechanisms for critical workflows to ensure timely issue detection and resolution.

Set up a foundation for scalable Data Model for the Stock Business

• Implement and build performant data models to solve common analytics use-cases at Stock.
• Knowledge Transfer and Documentation
• Thoroughly document the work performed, including methodologies, decisions, and any scripts or tools developed.
• Provide comprehensive knowledge transfer to the data team, ensuring a smooth transition and the ability to maintain the optimized data environment.

Requirements:

• 4 - 7 Years of Experience
• Proven experience in data engineering and data asset management.
• Proficiency in SQL, Python, and other relevant data processing languages and tools.
• Expertise in data modeling, ETL processes, and workflow orchestration (e.g., Airflow, Databricks).
• Strong analytical and problem-solving skills.
• Excellent communication and documentation abilities.
• Familiarity with cloud data platforms (e.g., Azure, AWS, GCP) is a plus.

For a confidential discussion or to find out more, contact Nitin Kuntiya on (909) 206-2199 or apply now.