Script to download from URL

[Kalinco]

Hi guys,
Up to now, I have been using a small python script to download files from the internet (mainly Dropbox) when given the URL of the file. (see below)
 

def download_file(url, download, desc=None):
if internet_on() == True:
u = urllib2.urlopen(url)

scheme, netloc, path, query, fragment = urlparse.urlsplit(url)
filename = os.path.basename(path)
if not filename:
filename = 'downloaded.file'
if desc:
filename = os.path.join(desc, filename)

with open(filename, 'wb') as f:
meta = u.info()
meta_func = meta.getheaders if hasattr(meta, 'getheaders') else meta.get_all
meta_length = meta_func("Content-Length")
file_size = None
if meta_length:
file_size = int(meta_length[0])
text = "Downloading: " + download
write(text.format(url, file_size),2)

file_size_dl = 0
block_sz = 8192
while True:
buffer = u.read(block_sz)
if not buffer:
break

file_size_dl += len(buffer)
f.write(buffer)

status = "{0:16}".format(file_size_dl)
if file_size:
status += " [{0:6.2f}%]".format(file_size_dl * 100 / file_size)
status += chr(13)
print(status, end="")
print()
return filename
else:
write("Internet Connection Lost, Please Check your Connection And Try Again",1)
time.sleep(5)
exit()


def internet_on():
try:
response = urllib2.urlopen('http://www.google.co.uk',timeout=5)
return True
except urllib2.URLError as err: pass
return False

 

However this method takes a very long time to download files that shouldn't take that long at all. Does anyone have a better, fast method of doing the same thing?

[Nonco]

So why would you want to use this over just downloading a file normally from Dropbox?

[slightlyjaded]

Hi guys,

Up to now, I have been using a small python script to download files from the internet (mainly Dropbox) when given the URL of the file. (see below)

 

def download_file(url, download, desc=None):

if internet_on() == True:

u = urllib2.urlopen(url)

scheme, netloc, path, query, fragment = urlparse.urlsplit(url)

filename = os.path.basename(path)

if not filename:

filename = 'downloaded.file'

if desc:

filename = os.path.join(desc, filename)

with open(filename, 'wb') as f:

meta = u.info()

meta_func = meta.getheaders if hasattr(meta, 'getheaders') else meta.get_all

meta_length = meta_func("Content-Length")

file_size = None

if meta_length:

file_size = int(meta_length[0])

text = "Downloading: " + download

write(text.format(url, file_size),2)

file_size_dl = 0

block_sz = 8192

while True:

buffer = u.read(block_sz)

if not buffer:

break

file_size_dl += len(buffer)

f.write(buffer)

status = "{0:16}".format(file_size_dl)

if file_size:

status += " [{0:6.2f}%]".format(file_size_dl * 100 / file_size)

status += chr(13)

print(status, end="")

print()

return filename

else:

write("Internet Connection Lost, Please Check your Connection And Try Again",1)

time.sleep(5)

exit()

def internet_on():

try:

response = urllib2.urlopen('http://www.google.co.uk',timeout=5)

return True

except urllib2.URLError as err: pass

return False

 

However this method takes a very long time to download files that shouldn't take that long at all. Does anyone have a better, fast method of doing the same thing?

Try typing the URL into one of the many free multistream download managers. I have had good experiences with DAP for almost a decade now.

 

If the problem isn't you, then this might fix it. If the problem is them....well it depends on how they limit their downloads.

[Kalinco]

So why would you want to use this over just downloading a file normally from Dropbox?

 

 

Try typing the URL into one of the many free multistream download managers. I have had good experiences with DAP for almost a decade now.

 

If the problem isn't you, then this might fix it. If the problem is them....well it depends on how they limit their downloads.

 

 

I am using this script to automate these downloads so typing into a download manager is not what I am looking for

[SpaceNugget]

curl or wget is what you are looking for.

[Gofspar]

Hi guys,

Up to now, I have been using a small python script to download files from the internet (mainly Dropbox) when given the URL of the file. (see below)

 

def download_file(url, download, desc=None):

if internet_on() == True:

u = urllib2.urlopen(url)

scheme, netloc, path, query, fragment = urlparse.urlsplit(url)

filename = os.path.basename(path)

if not filename:

filename = 'downloaded.file'

if desc:

filename = os.path.join(desc, filename)

with open(filename, 'wb') as f:

meta = u.info()

meta_func = meta.getheaders if hasattr(meta, 'getheaders') else meta.get_all

meta_length = meta_func("Content-Length")

file_size = None

if meta_length:

file_size = int(meta_length[0])

text = "Downloading: " + download

write(text.format(url, file_size),2)

file_size_dl = 0

block_sz = 8192

while True:

buffer = u.read(block_sz)

if not buffer:

break

file_size_dl += len(buffer)

f.write(buffer)

status = "{0:16}".format(file_size_dl)

if file_size:

status += " [{0:6.2f}%]".format(file_size_dl * 100 / file_size)

status += chr(13)

print(status, end="")

print()

return filename

else:

write("Internet Connection Lost, Please Check your Connection And Try Again",1)

time.sleep(5)

exit()

def internet_on():

try:

response = urllib2.urlopen('http://www.google.co.uk',timeout=5)

return True

except urllib2.URLError as err: pass

return False

 

However this method takes a very long time to download files that shouldn't take that long at all. Does anyone have a better, fast method of doing the same thing?

If in linux get wget or curl.

[Kalinco]

If in linux get wget or curl.

No i'm using Windows

[Kalinco]

curl or wget is what you are looking for.

I can't really use 3rd party programs, it needs to be using code that is in the default windows package or that can be converted to run within that environment

[SpaceNugget]

there is a windows equivalent to wget called invoke-webrequest

[Kalinco]

there is a windows equivalent to wget called invoke-webrequest

what program is that in? powershell?

[SpaceNugget]

what program is that in? powershell?

its in .net framework 3.5

 

https://msdn.microsoft.com/en-us/library/bb397419%28v=vs.90%29.aspx

[.:MARK:.]

You can get Wget for windows you know, it's pretty useful for batch scripts and so on.

[Cheesebaron]

Why isn't urllib enough for you?

import urlliburllib.urlretrieve ("http://www.example.com/songs/mp3.mp3", "mp3.mp3")

Maybe try out python-wget: https://bitbucket.org/techtonik/python-wget

[RhinoMcD]

curl or wget is what you are looking for.

 

You can get Wget for windows you know, it's pretty useful for batch scripts and so on.

 

What these guys said.

[Kalinco]

Thanks everyone for your help

 

I had tried;

urllib.urlretrieve ("http://www.example.com/songs/mp3.mp3", "mp3.mp3")

 

before however it did not work, therefore I was looking for an alternative solution. Then I worked out that the URL I was downloading from had expired so that was the reason it would not work.

 

Thanks anyway!