Sr. Compiler Engineer

Senior Compiler Engineer
Mountain View, CA
About DataPelago:

DataPelago is at the forefront of revolutionizing data processing for traditional analytics and cutting-edge GenAI preprocessing. We are building an innovative data processing engine that is transforming how Apache Spark, Apache Flink, Ray and others operate on diverse, large-scale data. Our team of engineers drive and adopt advances in hardware-accelerated computing, parallel processing of large-scale data, query optimization, distributed systems, compilers, machine learning, and cloud-native computing. We are looking for specialists to join our engineering team and shape the future of accelerated data processing.
The Opportunity:
As a Senior Compiler Engineer, you will lead the advancement of compilers for DataPelago’s data processing engine. You will enhance functional breadth, performance, scale, and reliability of DataPelago engine’s compilers in executing data processing workloads on diverse acceleration hardware infrastructure, including CPUs, GPUs. And FPGAs. This is a unique opportunity to make a significant impact on a category-defining product and work with a talented team of engineers.

What You'll Do:

• Architect: Lead the architecture of how our data processing engine will translate data pro-
cessing logical and physical plans into efficient code for execution on CPU/GPU/FPGA.

• Design: Lead design of functional and performance enhancements to the compilers in our
data processing engine.

• Core Development: Individually design, implement, test, optimize, and maintain compo-
nents of compilers for the data processing engine.

• Innovation and Differentiation: Analyze advances in the capabilities of CPU and GPU pro-
cessing elements from Nvidia, AMD, Intel, and others, compiler platforms and tools and

identify opportunities for our engine to enhance technology and product leadership.
• Collaboration: Partner effectively with engineering and product management in defining
language specifications of our product and ensuring compliance with industry standards,
platform definitions, and workload requirements.
• Continuous Improvement: Foster best practices in design and code reviews, testing, CI/CD,
and issue resolution to maintain highest product quality, security, efficiency, & productivity.
What You'll Bring:

• Bachelor's degree in Computer Science or a related field with 7+ years of relevant experi-
ence OR a Master's degree in Computer Science or a related field with 5+ years of relevant

experience.

• 5+ years of deep technical experience in developing and enhancing production-quality com-
pilers, tools, or related software.

• Demonstrated experience with auto-vectorization compiler technologies and data-parallel
architectures.

• Demonstrated experience with compiler internals. Solid experience with GCC/LLVM/
MLIR.

• Demonstrated experience with parsing, IR, type systems, and static analysis. Solid experi-
ence with flex/yacc/ANTLR.

• Experience with compilation for data applications (e.g., query compilers, query planners),
data processing languages (e.g., SQL, Python), acceleration hardware (e.g., Nvidia GPU),
and data processing engines (e.g., Apache Spark, Presto) preferred.
• Exceptional programming skills in C, C++. Rust experience preferred.
• Extensive development experience in Linux environments.
• Strong analytical and problem-solving skills with a passion for performance optimization.
Location Considerations:

We value face-to-face collaboration, but recognize that talent can be found anywhere. Our engineering team works at our headquarters in Mountain View, CA, at our India office in Hyderabad, and at remote locations. This specific position will be based at our headquarters in Mountain View, CA.
Why Join DataPelago?
• Technology Leadership: Shape the architecture and development of how our core engine
compiles parallel data processing applications on advanced acceleration hardware.
• Cutting-Edge Innovation: Work on challenging problems at the forefront of accelerated
computing and data processing.
• Significant Impact: Your contributions will directly impact the performance and scalability
of our mission-critical platform.
• Mentorship and Growth: Mentor and guide other talented engineers while expanding your
own technical expertise.

• Competitive compensation, stock options, comprehensive benefits package, leadership de-
velopment opportunities.