Core Data Engineer

Flow Traders is seeking a Core Data Engineer with a distinct focus on real-time analytics solutions. In this role, you will be responsible for owning the end-to-end lifecycle of our most critical data products that bring low-latency analytical capabilities to end users.You will be the architect and executor of the blueprint for these real-time data products, enhancing our Data Platform to develop and monitor automated pipelines. Your goal is to ensure that stakeholders across the organization have immediate access to reliable data whenever it is needed. You will further become an expert in these dynamic data sets—drawing from sources such as market data, network traffic and application logs—to optimize their use and explore additional sources for unlocking value in real time.

Collaboration is key, and you will consistently engage with teams across Flow Traders to ensure that our systems are designed with real-time data considerations at their core. Streamlining workflows related to data, and working with stakeholders, you will maximize the impactful insights drawn from real-time analytics.

As the steward of our most vital real-time data products, you will be uniquely positioned to influence Flow Traders' operations. Your accountability will lie in delivering these data products at exceptional quality—with a focus on low latency—to drive future growth. Ultimately, your work will provide substantial, real-time benefits felt across all divisions daily, laying one of the most crucial foundations for the firm's strategic expansion in the years ahead.

What you will do:

• Accountable for timely and high-quality delivery of core data products to the rest of the organization: Market Data, Trades/Orders/Quotes/Inquiries, Application Logs with a specific focus on low-latency analytics


• Develop and own the end-to-end lifecycle of these data products: build and maintain pipelines, develop quality checks, correct day-to-day issues, and systematize/automate manual processing steps


• Define and manage project plans to assume ownership of these data products


• Create and refine a common processing model (esp. fast vs slow path, use of alternative T3 solutions) across these data products


• Investigate currently available sources of data and ensure that Flow extracts maximum information and value from them


• Explore, evaluate, and onboard additional sources of available data (both independently and at the request of stakeholders)


• Work closely with other teams to: automate existing workflows, develop real-time capabilities as platform offering, ensure pipelines leverage standard Flow technologies, provide guidance on data-related architecture/design topics, ensure licensing and access control are appropriate

What you need to succeed:

• Excellent programming skills in Python


• Working proficiency with Linux, SQL and relational databases, and version control systems


• Proven experience with real-time OLAP databases like ClickHouse, Apache Pinot, StarRocks required


• Experience with Airflow, Kafka, and GCP (esp. GKE, Dataproc, and Big Query) highly desired


• Experience with Pandas, Spark, Apache Beam, Apache NiFi a plus


• Primarily technical, but domain-literate (exchange market data, network traffic data, security reference/pricing data, ETF compositions, application log data)


• Domain expertise a huge plus


• Extreme attention to detail, obsession with quality, perfectionist


• Innately driven to generalize, systematize, and automate


• Strong interpersonal skills, works very well with both technical and non-technical stakeholders


• Able to balance individual stakeholder demands in order to preserve quality and wide applicability of data


• Organized and able to define, manage, and execute projects (often several in parallel) while effectively triaging operational concerns

What We Offer

We like to think that talent grows at Flow and stays at Flow. To ensure this, we provide our employees with an extensive onboarding program, access to Flow Academy, the best working environment, the latest technology and continuous support. We go out of our way to retain the small business feeling with which we started and stimulate innovation and collaboration through teamwork and our non-hierarchical approach. We offer competitive salary, annual discretionary bonus and other fantastic perks and benefits, such as:

• Flow Academy for continuous learning and opportunities to attend domain-related conferences


• Comprehensive health insurance coverage


• In-house lounge with a bar, pool table and console games;


• Daily catered breakfast and lunch with healthy snacks and drinks available throughout the day;


• In-house hairdresser and massage therapist;


• Personal trainers, weekly boot camps and a subsidized gym membership;


• Annual company trip and a variety of events throughout the year


• and more!

Flow Traders does not accept unsolicited resumes from any professional staffing or search firms. All resumes, and any other information identifying potential candidates, submitted to any employee at Flow Traders via-email, the Internet or directly without a valid and signed search agreement will be deemed free to contact by Flow Traders without any restrictions and no placement fee of any kind will be paid in the event the candidate is hired by Flow Traders.