Python Program Not Working Correctly

[LtStaffel]

Hello everyone,

 

I am trying to write my own Python script to be given a link and find all links on that link, then find all links on those links, and so on. I believe it is called a scraper.

 

However, it is apparently not finding any links. The only code I've not written myself is the Regex URL finder string, which I imagine is probably the problem. I'm going to research and see if I can figure out if anything is wrong with it; but the code does not throw any errors. So I'm a bit lost.

 

import requests
import re
links = []
url = "LINK"
links.append(url)
for i in links:
    urli = requests.get(i)
    text = urli.text
    for j in text:
        link = re.findall('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', j)
        if type(link) == str:
            links.append(link)
            continue
        elif type(link) == list:
            for k in link:
                links.append(k)
            continue
        else:
            continue
    continue
print " ".join(links)

Thanks in advance.

[mtwest]

How levels deep are you hoping to go? Also it would be better to use a set rather than a list to store URLs as recursive linking situations could get you stuck in an infinite loop.

[fizzlesticks]

Don't use regex to parse html. Look at the BeautifulSoup or lxml libraries.

[Niemand]

On 27/11/2016 at 7:12 PM, mtwest said:

How levels deep are you hoping to go? Also it would be better to use a set rather than a list to store URLs as recursive linking situations could get you stuck in an infinite loop.

HyperloopText Markup Language

 

@LtStaffel what will you use that for?

[LtStaffel]

21 hours ago, Niemand said:

HyperloopText Markup Language

 

@LtStaffel what will you use that for?

To help find links on a website.

[Azgoth 2]

Like @fizzlesticks said, use the BeautifulSoup library.  Something like this:

 

import bs4
from urllib.requests import urlopen

page = bs4.BeautifulSoup(urlopen(whatever), "html.parser")
links = page.find_all("a")
links = [i['href'] for i in links]

Something like that should get you a list of all links on the page.