Crawler Development and maintenance: Mentor junior developers in crawler development techniques and best practices, Provide guidance in architectural decisions for data crawling and management.
Technical Leadership: Analyze and improve the performance of crawlers, reducing latency and increasing data retrieval efficiency, Address challenges related to rate limiting and blocking mechanisms, Performance Optimization: Implement security measures to protect sensitive data and ensure compliance with data privacy laws, Monitor and adapt to changes in social media platforms’s APIs, rules, and regulations
Security and Compliance: Integrate crawlers with existing systems and third-party tools to streamline data access and usage., Work closely with other engineering teams to incorporate new features and maintain compatibility across services.
Integration and collaboration: Build data pipelines to clean, process and store data effectively, Collaborate with data scientists and analysts to ensure data quality and availability for analysis.
Data Management: Design, develop and maintain robust web crawlers that can effectively collect data from Facebook, Tiktok, Threads, Linkedin, and other platforms, Optimize crawling algorithms to ensure data is gathered quickly and accurately while adhering to ethical standards and platforms policies
Job Requirement
Technical skills:
Strong programming skills in python, javascript, or the other relevant languages.
Hands-on experience with web scraping libraries and tools like beautiful soup, scrapy, selenium, or puppeteer.
Familiarity with API development and RESTful services.
Experience
Proven track record in building web crawlers for social media platforms
Understanding of platform-specific challenges (e.g., rate limiting, data structures,)
Data skills:
Experience working with big data tools (e.g., Hadoop, Spark).
