Senior Web Crawler Engineer

you.com is an AI-powered search and productivity platform designed to empower users with personalized, efficient, and trustworthy search experiences. As a cutting-edge technology company, we combine advanced AI models with user-first principles to deliver tools that enhance discovery, creativity, and productivity. At you.com, we are on a mission to create the most helpful search engine in the world—one that prioritizes transparency, privacy, and user control.

We’re building a team of innovators, problem-solvers, and visionaries who are passionate about shaping the future of AI and technology. At you.com, you’ll have the opportunity to work on impactful projects, collaborate with some of the brightest minds in the industry, and grow your career in an environment that values creativity, diversity, and curiosity. If you’re ready to make a difference and help us revolutionize the way people search and work, we’d love to have you join us!

you.com is an AI-powered search and productivity platform designed to empower users with personalized, efficient, and trustworthy search experiences. As a cutting-edge technology company, we combine advanced AI models with user-first principles to deliver tools that enhance discovery, creativity, and productivity. At you.com, we are on a mission to create the most helpful search engine in the world—one that prioritizes transparency, privacy, and user control.

We’re building a team of innovators, problem-solvers, and visionaries who are passionate about shaping the future of AI and technology. At you.com, you’ll have the opportunity to work on impactful projects, collaborate with some of the brightest minds in the industry, and grow your career in an environment that values creativity, diversity, and curiosity. If you’re ready to make a difference and help us revolutionize the way people search and work, we’d love to have you join us!

About the Role

As a Web Crawler Engineer at You.com, you will design, develop, and maintain automated systems to extract and index data from a wide variety of websites. Your work will directly power our search engine, ensuring users have access to the freshest and most comprehensive information available. You will collaborate with cross-functional teams to optimize crawling strategies, handle large-scale data, and ensure compliance with web standards and ethical guidelines 

This is a remote position open to candidates based in the United States. We also have co-working spaces in the San Francisco Bay Area and New York.

Responsibilities

• Design, implement, and maintain scalable web crawlers to collect and index web content efficiently and reliably.


• Develop robust scraping solutions to extract structured and unstructured data from diverse web sources, handling dynamic and complex websites


• Optimize crawling strategies for coverage, freshness, and efficiency, including intelligent scheduling and prioritization of web resources.


• Monitor and troubleshoot crawler performance, ensuring high availability and minimal downtime.


• Collaborate with data engineering and AI teams to integrate crawled data into downstream systems and search pipelines.


• Ensure compliance with robots.txt, site terms, and ethical web scraping practices.


• Stay up-to-date with the latest web technologies, anti-bot measures, and industry best practices.

Qualifications

• Strong programming skills in Python, Rust, C++, or similar languages.


• Familiarity with distributed systems, cloud infrastructure, and big data technologies.


• Experience with HTML parsing, and handling dynamically generated sites.


• Knowledge of anti-crawling techniques and strategies to mitigate them.


• Practical experience equivalent or equal to a BS/MS degree in Computer Science, Engineering, or a related field.


• Proven experience building and maintaining large-scale web crawlers or scraping systems is a plus

Company Perks:

• 
  

  Hubs in San Francisco and New York City offering regular in-person gatherings and co-working sessions

  
  


• 
  

  Flexible PTO with 11 U.S. holidays observed and a week shutdown in December to rest and recharge*

  
  


• 
  

  A competitive health insurance plan covers 100% of the policyholder and 75% for dependents*

  
  


• 
  

  12 weeks of paid parental leave in the US*

  
  


• 
  

  401k program, 3% match - vested immediately!*

  
  


• 
  

  $500 work-from-home stipend to be used up to a year of your start date*

  
  


• 
  

  $1,200 per year Health & Wellness Allowance to support your personal goals*

  
  


• 
  

  Chance to collaborate with a team at the forefront of AI research

  
  

*Certain perks and benefits are limited to full-time employees only

you.com is an E-Verify employer. We are also an inclusive, equitable, and accessible workplace. Please let us know if you require accommodation for any portion of the recruitment and hiring process.