webscraping football transfers

[Alesek]

Hello,

 

I am trying to webscrape https://www.transfermarkt.com/statistik/transfertage for football transfers using webscraper.io. The issue I am running into is how to set it up to scrape all days and all transfers per day as it shows only 25 transfers per page and the page number is shown only until 10 and then there is a new group of pages shown. Some days have 50+ pages of transfers 

 

Any ideas on how to do this?

 

[FlappyBoobs]

You're not the first to want this, search "transfermarkt api" and you'll get results like below. 

 

https://fcpython.com/blog/introduction-scraping-data-transfermarkt

[PAEz]

Cant you just target the next page link, Im assuming you targeted the individual links.
Something like this....

{"_id":"tryagain","startUrl":["https://www.transfermarkt.com/transfers/transfertagedetail/statistik/top/land_id_zu/0/land_id_ab/0/leihe//datum/2019-01-31"],"selectors":[{"id":"nextpage","type":"SelectorLink","parentSelectors":["_root","nextpage"],"selector":"li.naechste-seite a","multiple":false,"delay":0},{"id":"data","type":"SelectorText","parentSelectors":["_root","nextpage"],"selector":"td.hauptlink","multiple":true,"regex":"","delay":0}]}

This got all 43 pages of the target link.

If you want more help feel free to post your sitemap and Ill have a look at it.