Jump to content
Phishing Emails & YouTube Messages - Fake Giveaway Read more... ×
Search In
  • More options...
Find results that contain...
Find results in...
Alesek

webscraping football transfers

Recommended Posts

Posted · Original PosterOP

Hello,

 

I am trying to webscrape https://www.transfermarkt.com/statistik/transfertage for football transfers using webscraper.io. The issue I am running into is how to set it up to scrape all days and all transfers per day as it shows only 25 transfers per page and the page number is shown only until 10 and then there is a new group of pages shown. Some days have 50+ pages of transfers 

 

Any ideas on how to do this?

 


Threadripper 1950x

Gigabyte X399 Gaming 7

32GB DDR4 Corsair Vengeance

MSI GTX 1080Ti

Fractal Design Define S

Corsair RM1000i

Link to post
Share on other sites

Cant you just target the next page link, Im assuming you targeted the individual links.
Something like this....

{"_id":"tryagain","startUrl":["https://www.transfermarkt.com/transfers/transfertagedetail/statistik/top/land_id_zu/0/land_id_ab/0/leihe//datum/2019-01-31"],"selectors":[{"id":"nextpage","type":"SelectorLink","parentSelectors":["_root","nextpage"],"selector":"li.naechste-seite a","multiple":false,"delay":0},{"id":"data","type":"SelectorText","parentSelectors":["_root","nextpage"],"selector":"td.hauptlink","multiple":true,"regex":"","delay":0}]}

This got all 43 pages of the target link.

If you want more help feel free to post your sitemap and Ill have a look at it.

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×