Activate JobCopilot
On The Spot Development is a software development company that builds R&D teams for heavily invested startups and tech companies from the UK, EU, and Israel.
This job opportunity is for a position within one of our teams. You will be directly employed by On The Spot and will collaborate closely with our partner companies, contributing to their product's journey.
As a part of Orca cloud security suite, we are developing a product that processes massive amounts of real-time data. Our team drives Cloud Detection and Response (CDR) offering, managing large-scale API log ingestion, comprehensive stream and batch log analysis, and high-performance real-time services. By connecting software engineering, big data processing, cloud infrastructure and security, we are delivering scalable and reliable systems.
You will play a crucial role in designing, developing, and maintaining our real-time data processing infrastructure, which is going to become one of the key product offerings for the company.
This position requires deep technical expertise, ownership of major product domain and infrastructure components, and active collaboration with cross-functional teams, including product management, research, infrastructure, sales engineering, and customers.
Responsibilities:
- Own and drive the architecture, implementation, operation, and maintenance of the entire Cloud Detection and Response product offering, including breach detection, anomaly detection, and forensics
- Ensure the reliability, performance, and scalability of our infrastructure, making data-driven improvements.
- Develop and optimize real-time and batch processing pipelines for API log ingestion and analysis.
- Implement and enhance observability using metrics, logging, and tracing to monitor service health and performance.
- Work closely with product management and research teams to align technical decisions with business needs.
- Design and implement storage solutions optimized for different types of data (structured, semi-structured, unstructured).
Tech Stack:
- Programming Languages: Python, Go
- Frameworks: Django, FastAPI
- Messaging & Streaming: Kafka, SQS
- Infrastructure & Orchestration: Kubernetes, Docker
- Databases: PostgreSQL, SingleStore, Elasticsearch
- Monitoring & Observability: Grafana, Prometheus
- Version Control & CI/CD: GitHub
- Cloud infrastructure: AWS, GCP, Azure
- Analytics engines: Apache Spark
- Workflow management: Apache Airflow
Qualifications:
- 5+ years of experience in software engineering with a focus on backend, distributed systems, or real-time data processing.
- Strong expertise in building scalable and performant services.
- Experience designing and deploying real-time data pipelines using Kafka or similar technologies.
- Deep understanding of cloud infrastructure, preferably AWS, and containerized deployments using Kubernetes.
- Proficiency in working with databases (PostgreSQL, Elasticsearch, or similar) and optimizing data storage for high-scale environments.
- Strong problem-solving skills and a mindset for automation and efficiency.
- Excellent communication skills and the ability to work in cross-functional teams.
- Nice to have experience with Apache Spark & Apache Airflow
Bonus Points:
- Experience with real-time security and anomaly or threat detection systems.
- Familiarity with batch and stream processing frameworks.
- Prior work in high-scale, multi-tenant SaaS environments.
- Contributions to open-source projects related to real-time data processing.
Your email job alert is now active!
