AWS DATA ENGINEER & ETL EXPERT

We are looking for part time Freelancer for AWS Data Engineer Position.

Releavant Exp : 9+

Mandatory Tech Stack required: 

Source: Salesforce (ECRM & OSC)
Backup: Grax
Cloud: AWS (EC2, S3, CloudWatch, DynamoDB, RDS, Secrets Manager, ALB, ASG)
Infrastructure as Code: Terraform
File Format: Parquet
Target: Enterprise Data Lake (EDL)
Schema: Blue Schema
Database: Blue Database
Query Language: SQL

JD :

1. Build and maintain AWS data pipelines

* Develop ETL/ELT pipelines using AWS Glue, PySpark, Python, and SQL.
* Ingest data from sources like Salesforce, databases, APIs, and S3.
* Load curated data into Amazon Redshift and data lake storage.

2. Optimize Athena and data lake performance

* Convert JSON/CSV data into Parquet.
* Use Snappy compression.
* Design proper partitioning strategies.
* Resolve split limit and performance issues.
* Optimize Athena query costs.

3. Manage modern data lake architecture

* Work with Apache Iceberg tables.
* Perform migrations from traditional Parquet tables.
* Support schema evolution, time travel, and ACID transactions.

4. Production support and troubleshooting

* Investigate Glue jobs that suddenly become slow.
* Debug Lambda timeouts.
* Fix missing records and data quality issues.
* Resolve Redshift performance problems.
* Perform root cause analysis (RCA).

5. Infrastructure as Code

* Build AWS infrastructure using Terraform.
* Create reusable modules.
* Manage Auto Scaling Groups, ALBs, IAM, Lambda, Secrets Manager, S3, Redshift, and DynamoDB.
* Troubleshoot Terraform state and production deployment issues.

6. Security

* Manage AWS Secrets Manager.
* Configure Lambda-based secret rotation.
* Ensure Terraform does not overwrite rotated passwords.
* Implement IAM least-privilege access.

7. Backup and Disaster Recovery

* Work with enterprise backup tools like Rubrik (their environment may use Grax for Salesforce).
* Validate backup jobs.
* Perform restores.
* Support disaster recovery testing.
* Verify restored data.

8. Data Quality

* Validate source and target record counts.
* Maintain audit/control tables (for example in DynamoDB).
* Check checksums and duplicate records.
* Troubleshoot data discrepancies reported by business users.

9. AWS Services

* S3
* Glue
* Athena
* Lambda
* Redshift
* DynamoDB
* EventBridge
* Step Functions
* CloudWatch
* Secrets Manager
* IAM
* Auto Scaling Groups
* Application Load Balancer