caferemote
PortPro

Director of Software Engineering (Node.js & Web Scraping Expert)

Remote United States Full-time Lead Posted 3 days ago
Apply now
Work from
United States
Contract
Full-time
Experience
Lead
Posted
17 November 2026
Apply by
17 December 2026

We are seeking a Director of Software Engineering with deep expertise in Node.js development and large-scale web scraping. This role will lead the engineering team, designing and optimizing high-performance, distributed web scraping systems.

The ideal candidate has extensive experience in handling anti-bot measures, data pipeline optimization, and scalable cloud-based architectures.

Responsibilities

Key Responsibilities- Software Engineering & Web Scraping Leadership

  • Architect, develop, and maintain scalable and distributed web scraping systems using Node.js.
  • Design and implement data extraction pipelines to process large volumes of structured and unstructured data.
  • Develop solutions to bypass anti-bot mechanisms, including CAPTCHA handling, session management, fingerprinting, and IP rotation.
  • Optimize scraping processes for performance, reliability, and efficiency while managing proxy services(residential, datacenter, rotating).Oversee data storage and processing strategies, ensuring high availability and consistency.
  • Collaborate with Product, DevOps, and Data Science teams to integrate extracted data into analytics and business applications.
  • Implement best practices for microservices, API integrations, and real-time data streaming.

Key Responsibilities- Scalability, Security & DevOps

  • Lead the transition to cloud-native, containerized, and serverless architectures for web scraping.
  • Ensure compliance with legal and ethical standards (robots.txt, GDPR, CCPA, etc.).Optimize cloud resources (AWS, GCP, or Azure) to support high-throughput scraping.
  • Manage real-time monitoring and alerting systems to detect scraping failures, IP bans, or performance bottlenecks.
  • Work closely with DevOps teams to optimize CI/CD pipelines, automated deployments, and system scalability.

Key Repsonsibilities- Engineering Team Management & Strategy

  • Lead, mentor, and grow a high-performance engineering team.
  • Define and execute the technology roadmap, aligning with business objectives.
  • Foster a culture of continuous learning, collaboration, and innovation.
  • Implement agile development methodologies (Scrum, Kanban) to optimize project execution.
  • Ensure code quality, security, and best practices across all engineering efforts.

Requirements

Qualifications & Experience- Technical Expertise

  • 10+ years of experience in software engineering, with at least 5+ years in web scraping and large-scale data extraction.
  • Strong hands-on expertise in Node.js, Puppeteer, Playwright, Cheerio, Selenium, and headless browser automation.
  • Extensive experience in handling CAPTCHAs, IP rotation, session management, and anti-bot evasion techniques.
  • Deep knowledge of proxy management (residential, datacenter, rotating, and VPNs).Experience with NoSQL/SQL databases (MongoDB, PostgreSQL, Redis, Elasticsearch, etc.).
  • Familiarity with data processing frameworks (Kafka, RabbitMQ, Spark, Airflow, etc.).Strong experience with CI/CD, containerization (Docker, Kubernetes), and cloud deployment (AWS/GCP/Azure).

Qualifications & Experience- Leadership & Soft Skills

  • Proven track record of scaling engineering teams and leading complex projects.
  • Strong problem-solving and debugging skills, especially for scraping challenges and performance bottlenecks.
  • Excellent communication and stakeholder management skills.
  • Passion for mentorship, team development, and continuous learning.

Nice to have

  • Experience with machine learning for data extraction and NLP.
  • Knowledge of browser fingerprinting and bot detection mechanisms.
  • Familiarity with enterprise-scale web crawling frameworks (Scrapy, Colly, Apify, etc.).
  • Prior leadership experience in data-driven businesses or web scraping startups.

Discussion

No comments yet. Ask a question or share a tip about this role.

Apply now

Similar remote jobs

Browse all →

Senior AI Agent Engineer

Bonapolia
Remote Albania, Andorra, Armenia, Austria, Belarus, Belgium, Bosnia and Herzegovina, Bulgaria, Croatia, Cyprus, Czechia, Denmark, Estonia, Faroe Islands, Finland, France, Georgia, Germany, Gibraltar, Greece, Guernsey, Holy See (Vatican City State), Hungary, Contract

We are looking for a Senior AI Agent Engineer: 🗣 Language Proficiency: Upper-Intermediate 🧾 Employment type: Full time 🌍 Candidate Location: Poland, other EU countries, Kazakhstan,…

5h ago Back-End Programming

Staff Software Engineer - Golang

Rapid7
Remote United States Full-time $164K – $222K / year

Rapid7 is seeking a Staff Software Engineer, AI-SOC to own the control plane and API server that expose our autonomous investigation platform to more than 10,000 customers. This role is for…

5h ago Back-End Programming

Cloud & Backend Software Engineer

UIC Alaska
Remote United States Full-time $110K – $130K / year

Overview CLOUD & BACKEND SOFTWARE ENGINEER (ERDC AI-DRIVEN ENGINEERING QA/QC): Bowhead is seeking a Cloud & Backend Software Engineer to support the development of an AI-driven Engineering…

7h ago Back-End Programming

Software Engineer III

MCCi
Remote United States Full-time

About Us: MCCi is a premier provider of intelligent business process automation solutions and a trusted partner to over 2,000 organizations across state and local government, education, and…

9h ago Back-End Programming

Backend Engineer

Domaine
Remote Canada Full-time

Meet Domaine , a global digital commerce and transformation partner for ambitious and beloved brands. We are the world’s largest independent Shopify partner. We are also a growing team of…

10h ago Back-End Programming

Senior Software Reliability Engineer – Splunk Framework & Runtime Services

Cisco
Remote United States Full-time $139K – $203K / year

The application window is expected to close on: 11/27/2026 Note: This role is Remote in the US with a strong preference for locations except CA, WA, DC, MD, NJ, NY, CT, MA, VA, AK. Meet the…

11h ago Back-End Programming

Get remote jobs the moment they are posted

Choose your categories and skills. We push matching jobs to your device. No email, ever.