Data Engineer III

The Enterprise Next-Gen Big Data Engineering team works horizontally with all use-case and business teams across the organization. Rather than owning a single application or business domain, the team identifies optimization opportunities, establishes engineering standards, designs reusable frameworks, and creates common platform capabilities that accelerate development and improve operational excellence within the American Express Big Data ecosystem on Google Cloud Platform (GCP).

We are seeking a highly experienced and hands-on GCP Data Platform Engineer to improve engineering effectiveness, performance, scalability, and reliability of data platforms within the American Express Big Data ecosystem on GCP.

The ideal candidate has a strong foundation in GCP data services, distributed data processing, BigQuery, Apache Spark, Python, Airflow and software engineering, combined with practical experience building APIs and platform services using technologies such as Spring Boot and FastAPI on GCP.

Deep expertise in a specific business domain is not required. We are looking for engineers who possess strong fundamentals in GCP-native data engineering, can quickly understand unfamiliar systems, identify bottlenecks and architectural weaknesses, and design pragmatic solutions that can be adopted across multiple teams. Strong conceptual understanding of GenAI, RAG, LLM applications, and agentic systems is required, with hands-on experience being a strong plus.

• Partner with use-case and business teams to understand existing data platforms, workloads, and operational challenges; identify opportunities to improve performance, scalability, reliability, and cost-efficiency.
• Analyze and optimize large-scale Apache Spark and PySpark workloads, including joins, shuffles, partitioning, skew, caching, memory utilization, executor configuration, and resource allocation, and diagnose performance problems across distributed compute, storage, and orchestration layers.
• Develop systematic approaches, tooling, and measurable baselines for identifying and evaluating optimization opportunities across throughput, latency, resource utilization, and cost.
• Optimize GCP platform costs by reviewing and improving code, queries, resource configurations, and architectural patterns across BigQuery, Dataproc, Dataflow, Cloud Composer, and other GCP services.
• Design and develop reusable engineering frameworks, libraries, APIs, and platform components for ingestion, transformation, validation, observability, error handling, retries, reconciliation, and data quality, adoptable across multiple teams.
• Define engineering standards, reference architectures, and best practices for batch processing, streaming (Pub/Sub, Dataflow), event-driven processing, and workflow orchestration (Cloud Composer, Astronomer, Apache Airflow).
• Establish standards around logging, monitoring, alerting, tracing, CI/CD, automated testing, deployment automation, and production readiness.
• Conduct architecture and performance reviews across teams; identify recurring problems and convert them into reusable platform capabilities.
• Build proofs of concept and reference implementations to evaluate new technologies, frameworks, and optimization approaches; partner with architecture teams to influence technology direction.
• Mentor engineers, create technical documentation and engineering playbooks, and drive adoption of standards and best practices across the organization

• Strong conceptual understanding of LLM-powered applications, Retrieval-Augmented Generation (RAG), agentic AI, and multi-agent architectures, including their data requirements and pipeline needs.

• Ability to design and build data engineering pipelines that feed GenAI processes, covering ingestion, chunking, embeddings, vector search, hybrid retrieval, and relevance optimization.

• Familiarity with Vertex AI, GenAI evaluation (groundedness, hallucination detection), and GenAI security considerations (prompt injection, PII handling, guardrails).

• Ability to quickly understand unfamiliar architectures, applications, and technology stacks.

• Ability to work through ambiguous engineering problems where the solution may not be known upfront.

• Experience mentoring engineers and driving adoption of engineering standards and best practices.

• Experience entering unfamiliar systems, identifying performance or scalability problems, and converting solutions into reusable platform capabilities.

• Experience evaluating architectural and distributed-system trade-offs across latency, throughput, scalability, consistency, reliability, operational complexity, and cost.