Lead Software Engineer – Scraping

• Practice Development & Integration

• Technologies Development Skills

• Location New York, United States

• Type Permanent

Role:

Our client is an innovative FinTech company and the market leader in providing data and business intelligence to help customers make faster and smarter investment decisions. They are looking for a Lead Software Engineer to drive and own theit web scraping initiatives. This role will lead the design, development, and maintenance of scalable scraping solutions as part of their data technology team. The ideal candidate is an expert Python developer with strong experience in web scraping and related technologies.

Responsibilities

• Provide effective technical leadership and hands-on technical mentorship to developers.
• Lead the design and development of web scraping applications and services, ensuring they are scalable, efficient, and meet performance requirements.
• Drive technical decisions, ensuring that best practices are followed.
• Provide leadership in applying design patterns and adhering to SOLID engineering principles, with a focus on reusability, modularity, scalability, and fault tolerance.
• Utilize hands-on experience to design, build, and support web scraping solutions, particularly using Python and relevant libraries (e.g., Scrapy).
• Promote a culture of continuous improvement and high-quality code delivery.
• Partner with the broader technology leadership team to ensure service reliability through testing and release procedures.
• Stay current with industry trends in web scraping, data technologies, and emerging tools.
• Identify opportunities for innovation and process improvement,
• Ensure compliance with security and firmwide architecture standards.

Requirements

• Strong understanding of technical design patterns and SOLID engineering principles.
• Experience mentoring engineering teams, particularly in the context of data extraction and scraping projects.
• Expertise in Python development, particularly with web scraping libraries like Scrapy/Beautiful Soup.
• Proven experience in developing and maintaining large-scale web scraping solutions.
• Familiarity with C#/.NET technologies is a strong plus.
• Strong problem-solving skills and a proactive attitude.
• Strong understanding of API design principles and best practices
• Experience with CI/CD pipelines and automated testing frameworks.
• Familiarity with cloud platforms (AWS/GCP/Azure)
• Experience with software development lifecycle tools such as JIRA, Confluence, GitHub, etc.
• Excellent communication and collaboration skills.

This position offers comprehensive benefits and is eligible for an annual discretionary bonus.

Fully remote candidates will be considered.

Cannot provide Visa sponsorship at this time.

• share
• 
• 
• 
•