Data Engineer II

Data Engineer II

Reports to: Team Lead

Overview:

The Data Engineer II will architect and implement data systems at TruBase. This role focuses on designing distributed data processing systems, optimizing data pipelines, and ensuring system reliability at scale.

Essential Behaviors & Activities:

1. Build distributed data systems
   
   
   
   1. Design for fault tolerance Balance consistency and availability requirements
   
   
   2. Scale processing across multiple instances
   
   
   
   


2. Implement database solutions
   
   
   
   1. Recommend appropriate database technologies based on workloads
   
   
   2. Design efficient schemas and indices
   
   
   3. Optimize query patterns and data access
   
   
   
   


3. Build efficient and correct ETL pipelines
   
   
   
   1. Design high-throughput data flows
   
   
   2. Implement data transformation logic
   
   
   3. Monitor pipeline performance and correctness
   
   
   4. Ensure data integrity across systems
   
   
   
   


4. Troubleshoot performance issues
   
   
   
   1. Profile system bottlenecks
   
   
   2. Analyze memory, CPU, and network usage
   
   
   3. Implement distributed tracing
   
   
   
   


5. Advance technical capabilities
   
   
   
   1. Be familiar with emerging technologies
   
   
   2. Implement proof-of-concepts
   
   
   3. Document architectural decisions
   
   
   4. Mentor junior engineers
   
   
   
   

Competencies and Experience:

Core Competencies

• Distributed systems knowledge (idempotence, consistency, consensus)


• Understand how databases work (B+-trees vs LSMs, columnar vs. row storage, storage tiering)


• Performance debugging and profiling practices and tools (strace, py-spy, flamegraphs)


• Memory access/usage, IO, and CPU usage optimization


• Data transformation and aggregation at scale


• Experience with log-based distributed streaming systems (Kafka, Pulsar, Kinesis)


• Experience with relational and columnar databases (Postgres, Redshift, Clickhouse, Dynamo)


• Advanced language skills (Python, TypeScript)


• Container orchestration (Kubernetes)


• CI/CD for data pipelines

Desirable Technical Experience

• Experience with lower level lang


• Understanding of codecs and compression (JSON, Protobuf, FlatBuffers)


• AWS cloud architecture


• Data governance implementation

Measurables

• System reliability and scalability


• Query and pipeline performance improvements


• Architecture quality


• Technical documentation clarity


• Knowledge sharing effectiveness

Other Information:

Supervisory Responsibility: This position has no direct reports.

Position Type/Expected Hours of Work

• This is a full-time position; days and hours of work are Monday through Friday, 40 hours a week.


• (Hybrid, Atlanta Area — 2 Days/Week in Office)


• Work Authorization:Applicants must be legally authorized to work in the United States for any employer. We are unable to provide visa sponsorship (e.g., H-1B, OPT extensions) at this time.

This job description is not designed to cover or contain a comprehensive listing of activities, duties or responsibilities that are required of the employee for this job. Duties, responsibilities and activities may change at any time with or without notice.