Skip to content

Optimize the scraper #3

@Mahmoudgalalz

Description

@Mahmoudgalalz

problem definition

now the job of running the scrapper take around ~2h to be finished this is huge time and we can decrease that time

solution

we need to run the max number of job separately as we have 20 job to by running in github actions

as we have many search queries in both sites

so I propose to create a cli tool that help us which task will run to apply that to github action to run concurrent jobs

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions