Hoster charging 75gbp to migrate my website.

[The Torrent]

So my website is with LCN.

 

I tried downloading my website using WinSCP but its 40gb and the download speed on WinSCP is 50kbps.... obviously you can tell this is not viable.

 

I phoned up LCN, they said they will do it for a fee of... 75GBP! And it takes 3-4 weeks.

 

Is there actually no other way for me to go but to pay a bunch of money to migrate my website???

[Karbust]

Instead of downloading to your PC so you upload to the new host, just make a transfer over SFTP and remove yourself from the middle. On Debian and FreeBSD you can use the SCP command to such thing (the only OS's I have used).

 

This is a script I made a while ago to transfer backups to remote locations, you can make the necessary changes to work for your use case:

#!/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin

# Bins
SSH=`which ssh`
SCP=`which scp`
 
# Date for folders and filenames
DAY=$(date +"%Y-%m-%d.%T")
 
# Local backup folder (no trailing slash)
LOCAL_FOLDER="/tmp/backups"
 
# SFTP Configuration
REMOTE_HOST="IP"
REMOTE_PORT="PORT"
REMOTE_USER="USER"
REMOTE_PEM="/PATH/TO/KEY" #With no trailing slash
REMOTE_FOLDER="/PATH/TO/DESTINATION/FOLDER/" # With trailing slash

# Create the remote folder
$SSH -p "${REMOTE_PORT}" -i "${REMOTE_PEM}" "${REMOTE_USER}@${REMOTE_HOST}" "mkdir ${REMOTE_FOLDER}${DAY}"
# Transfer the files to the remote folder
$SCP -P "${REMOTE_PORT}" -r -i "${REMOTE_PEM}" "${LOCAL_FOLDER}" "${REMOTE_USER}@${REMOTE_HOST}:/${REMOTE_FOLDER}/${DAY}"

# Delete local dump files
rm -f $LOCAL_FOLDER/*

 

If you don't need to use a private key for authentication, just remove this:

-i "${REMOTE_PEM}"

 

Hope it helps.

[The Torrent]

Just now, Karbust said:

Instead of downloading to your PC so you upload to the new host, just make a transfer over SFTP and remove yourself from the middle. On Debian and FreeBSD you can use the SCP command to such thing (the only OS's I have used).

 

This is a script I made a while ago to transfer backups to remote locations, you can make the necessary changes to work for your use case:

#!/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin

# Bins
SSH=`which ssh`
SCP=`which scp`
 
# Date for folders and filenames
DAY=$(date +"%Y-%m-%d.%T")
 
# Local backup folder (no trailing slash)
LOCAL_FOLDER="/tmp/backups"
 
# SFTP Configuration
REMOTE_HOST="IP"
REMOTE_PORT="PORT"
REMOTE_USER="USER"
REMOTE_PEM="/PATH/TO/KEY" #With no trailing slash
REMOTE_FOLDER="/PATH/TO/DESTINATION/FOLDER/" # With trailing slash

# Create the remote folder
$SSH -p "${REMOTE_PORT}" -i "${REMOTE_PEM}" "${REMOTE_USER}@${REMOTE_HOST}" "mkdir ${REMOTE_FOLDER}${DAY}"
# Transfer the files to the remote folder
$SCP -P "${REMOTE_PORT}" -r -i "${REMOTE_PEM}" "${LOCAL_FOLDER}" "${REMOTE_USER}@${REMOTE_HOST}:/${REMOTE_FOLDER}/${DAY}"

# Delete local dump files
rm -f $LOCAL_FOLDER/*

 

If you don't need to use a private key for authentication, just remove this:

-i "${REMOTE_PEM}"

 

Hope it helps.

apparently i cant do this because i need to upload the file in zip format, which means i need to dl it locally first then use 7zip to zip it.

 

Also i use windows so that script is a bit over my head.

 

[Bombastinator]

15 minutes ago, The Torrent said:

So my website is with LCN.

 

I tried downloading my website using WinSCP but its 40gb and the download speed on WinSCP is 50kbps.... obviously you can tell this is not viable.

 

I phoned up LCN, they said they will do it for a fee of... 75GBP! And it takes 3-4 weeks.

 

Is there actually no other way for me to go but to pay a bunch of money to migrate my website???

Is there a dial up modem involved with this anywhere?  That’s some really slow transfer

[The Torrent]

Just now, Bombastinator said:

Is there a dial up modem involved with this anywhere?  That’s some really slow transfer

no lol idk why its so slow i think its just because theres thousands of small files but still....

 

i cant pause a winSCP transfer either as far as i know, so if this isn't done by the end of the day when i gotta shut down my pc its been a waste of time.

 

[Karbust]

2 minutes ago, The Torrent said:

apparently i cant do this because i need to upload the file in zip format, which means i need to dl it locally first then use 7zip to zip it.

 

Also i use windows so that script is a bit over my head.

 

That script is to run on the server you have the "origin" code, you only need your computer to open an SSH shell to run the script, you can even put it on a "screen" so you can close it.

 

Your servers use Windows? If yes just install WinSCP on the either side (lets say on the origin) and an SFTP server on the opposite side (let's say the destination).

 

The speed you mentioned, it's on the servers or on your personal network? Because if it's on either server good luck doing anything without having to pay the fee you mentioned.

[The Torrent]

4 minutes ago, Karbust said:

That script is to run on the server you have the "origin" code, you only need your computer to open an SSH shell to run the script, you can even put it on a "screen" so you can close it.

 

Your servers use Windows? If yes just install WinSCP on the either side (lets say on the origin) and an SFTP server on the opposite side (let's say the destination).

 

The speed you mentioned, it's on the servers or on your personal network? Because if it's on either server good luck doing anything without having to pay the fee you mentioned.

no i dont have a server.

im trying to download the website from LCN servers, yh so i think your last paragraphs applies to me........... i may have to sleep with my pc on for a few days even though its right next to my bed...

[Karbust]

4 minutes ago, The Torrent said:

no i dont have a server.

im trying to download the website from LCN servers, yh so i think your last paragraphs applies to me........... i may have to sleep with my pc on for a few days even though its right next to my bed...

When I have downloads/uploads to do on my PC I just leave it on, it's 2 meters away from my bed, just put the fans at the lowest speed, the CPU is barely doing anything anyway, so they don't need to be on full blast.

[The Torrent]

1 minute ago, Karbust said:

When I have downloads/uploads to do on my PC I just leave it on, it's 2 meters away from my bed, just put the fans at the lowest speed, the CPU is barely doing anything anyway, so they don't need to be on full blast.

my biggest issue is i decided to download to my hard drive for some stupid reason ffffff, but yh otherwise i might just put a fan curve of all fans off until like 60c then they go on quietly. i got a top mounted rad so it should passively cool my cpu enough hopefully...

[mariushm]

Use Filezilla FTP client, set all transfers to binary, set 10 simultaneous transfers and download your files. 

Small files do download slow, but as it downloads 10 files in parallel, it transfers much faster. 

 

Alternatively, you could upload a script that makes a TAR or ZIP of a folder and then you only need to download that folder?

 

If your electricity is good/reliable and your laptop's battery is solid, you could set up a ram drive on your laptop and download data into ram drive - ex make a 8-14 GB ram drive (your laptop's ram amount - a few GB), as it gets filled, move completed files to the hard drive. This way your hard drive sleeps most of the time so your laptop isn't noisy.

 

[The Torrent]

4 minutes ago, mariushm said:

Use Filezilla FTP client, set all transfers to binary, set 10 simultaneous transfers and download your files. 

Small files do download slow, but as it downloads 10 files in parallel, it transfers much faster. 

 

Alternatively, you could upload a script that makes a TAR or ZIP of a folder and then you only need to download that folder?

 

If your electricity is good/reliable and your laptop's battery is solid, you could set up a ram drive on your laptop and download data into ram drive - ex make a 8-14 GB ram drive (your laptop's ram amount - a few GB), as it gets filled, move completed files to the hard drive. This way your hard drive sleeps most of the time so your laptop isn't noisy.

 

you can do simultaneous transfers in filezilla? i was told to avoid filezilla because it has adware or something in it, so went WinSCP instead.

 

Currently the speed increased slightly to about 1mb/s average. estimated time left is 18 hours and its a 39gb website. Do you think i should cancel this and try filezilla? Or just commit to this now?

[mariushm]

Sourceforge may bundle some toolbar inside the setup if you download through Sourceforge or using the big green button that says "download filezilla ftp client" on the filezilla page. That setup has "sponsored" in the file name.

 

If you go on the filezilla page, there's a "show additional download options" link where it's a download link for un-altered, no-adware included setup : https://filezilla-project.org/download.php?show_all=1

 

You can configure up to 10 parallel transfers, but nothing stops you from opening two or more instances of filezilla ftp client , you just have to be careful to set each one to download a different folder or set of files. Then, each program will do up to 10 simultaneous transfers 

 

As long as the remote server lets you have that many simultaneous connections.

[Karbust]

30 minutes ago, The Torrent said:

you can do simultaneous transfers in filezilla? i was told to avoid filezilla because it has adware or something in it, so went WinSCP instead.

 

Currently the speed increased slightly to about 1mb/s average. estimated time left is 18 hours and its a 39gb website. Do you think i should cancel this and try filezilla? Or just commit to this now?

I personally love using Cyberduck for bigger files (it also allows simultaneous file transfers), I find it much more responsive and "faster" than FileZilla, but for day to day usage, I use FileZilla, it doesn't have any adware, if you download from the official website.

[The Torrent]

41 minutes ago, mariushm said:

Sourceforge may bundle some toolbar inside the setup if you download through Sourceforge or using the big green button that says "download filezilla ftp client" on the filezilla page. That setup has "sponsored" in the file name.

 

If you go on the filezilla page, there's a "show additional download options" link where it's a download link for un-altered, no-adware included setup : https://filezilla-project.org/download.php?show_all=1

 

You can configure up to 10 parallel transfers, but nothing stops you from opening two or more instances of filezilla ftp client , you just have to be careful to set each one to download a different folder or set of files. Then, each program will do up to 10 simultaneous transfers 

 

As long as the remote server lets you have that many simultaneous connections.

allllright you gave me an idea, i cancelled my download and half the directory is made with winSCP. now im running multiple winSCP instances going like foder by folder, choosing not to overwrite and its copying in whatevers missing. this should be faster because im doing multiple at once.

[mariushm]

2 minutes ago, The Torrent said:

allllright you gave me an idea, i cancelled my download and half the directory is made with winSCP. now im running multiple winSCP instances going like foder by folder, choosing not to overwrite and its copying in whatevers missing. this should be faster because im doing multiple at once.

You should choose RESUME  because when you cancelled WinSCP the first time you probably had an incomplete transfer and when you choose to not overwrite the second run, it probably left that single file incomplete. 

[Bombastinator]

1 hour ago, The Torrent said:

no lol idk why its so slow i think its just because theres thousands of small files but still....

 

i cant pause a winSCP transfer either as far as i know, so if this isn't done by the end of the day when i gotta shut down my pc its been a waste of time.

 

That wouldn’t do it.  There would be some overheard, but it wouldn’t be THAT bad.  There is something up.  At that kind of speed 40gb would take weeks.

[The Torrent]

36 minutes ago, mariushm said:

You should choose RESUME  because when you cancelled WinSCP the first time you probably had an incomplete transfer and when you choose to not overwrite the second run, it probably left that single file incomplete. 

omg thats a good point.

 

uhhh WHOOPS in that case i have messed up.................................... because since my last message ive done this a lot with like 10 instances running....slowly breaking down folders more and more and more... you have a solid point about that last file incomplete... crap.

 

what do you think the best way around this is? do i have to do it alllll over???

[The Torrent]

24 minutes ago, Bombastinator said:

That wouldn’t do it.  There would be some overheard, but it wouldn’t be THAT bad.  There is something up.  At that kind of speed 40gb would take weeks.

ok so since that message i decided to break down folders and am running a bunch of winSCP instances at once.

 

So most of them run in the kbps but there was this one folder called ai1wm-backups thats like 15gb big on its own, and that currently is being downloaded at like 10 MB/S which is my maximum speed... so thats good ig but also means the reason it was slow earlier must be because of thousands of small files.

[Bombastinator]

3 minutes ago, The Torrent said:

ok so since that message i decided to break down folders and am running a bunch of winSCP instances at once.

 

So most of them run in the kbps but there was this one folder called ai1wm-backups thats like 15gb big on its own, and that currently is being downloaded at like 10 MB/S which is my maximum speed... so thats good ig but also means the reason it was slow earlier must be because of thousands of small files.

Or that you worked around the issue.  I could see even 100% overhead but that kind of massive difference is more like 1000% it could be the big file is coming in UDP or something because it’s large and is missing the bottleneck.

[mariushm]

28 minutes ago, The Torrent said:

omg thats a good point.

 

uhhh WHOOPS in that case i have messed up.................................... because since my last message ive done this a lot with like 10 instances running....slowly breaking down folders more and more and more... you have a solid point about that last file incomplete... crap.

 

what do you think the best way around this is? do i have to do it alllll over???

Download the files again, with the RESUME option ... incomplete files will be resumed and completed,  complete files will be skipped altogether.

If WinSCP has no resume feature, use FTP. 

 

Alternatively, if you can run scripts or something on the server, you could run a script to calculate a MD5 checksum or CRC32 for each file, and then you can run the same script on the local computer and compare the checksums and you'll spot right away the differences and transfer those incomplete files.

 

 

[The Torrent]

2 minutes ago, mariushm said:

Download the files again, with the RESUME option ... incomplete files will be resumed and completed,  complete files will be skipped altogether.

If WinSCP has no resume feature, use FTP. 

 

Alternatively, if you can run scripts or something on the server, you could run a script to calculate a MD5 checksum or CRC32 for each file, and then you can run the same script on the local computer and compare the checksums and you'll spot right away the differences and transfer those incomplete files.

 

 

im confused.

 

WinSCP has no resume feature as far as i know.

 

Not sure what you mean by FTP. i thought i am using FTP? What else can i use? It says FTP in the bottom right of the winSCP window.

[mariushm]

First you have to figure out what exactly you are using... FTP , SFTP, FTPS etc etc 

 

WinSCP supports multiple protocols : FTP, SFTP (SSH FTP,  which is different than FTP, not same thing : https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol ) , SCP (secure copy protocol : https://en.wikipedia.org/wiki/Secure_copy_protocol) 

 

Filezilla FTP Client supports FTP, FTPS (FTP over TLS, or FTP with security extensions : https://en.wikipedia.org/wiki/FTPS  - basically commands you give to remote computer and transfers are encrypted, but it's still FTP) , and SFTP. 

 

FTP and FTPS supports resume, IF the remote server allows resume. 

SFTP and SCP may or may not allow resume, I'm not sure. 

 

It's up to the application you use if it supports resume or not. Maybe WinSCP doesn't do resume with the mode you're using.

The remote server may also refuse to provide resume support, though it's rare these days. 

 

[Bombastinator]

11 hours ago, The Torrent said:

im confused.

 

WinSCP has no resume feature as far as i know.

 

Not sure what you mean by FTP. i thought i am using FTP? What else can i use? It says FTP in the bottom right of the winSCP window.

There is FTP (File Transfer Protocol) and UDP (User Datagram Protocol). UDP is generally used for transferring big stuff.  It’s fasterpartially  because it doesn’t require a handshake. As such it doesn’t have the error correction that FTP does, which doesn’t matter as much for things like photos. Both go over tcp/ip.  If you’re thinking at this point that there are perhaps too many acronyms you are correct.  Also you have no idea how bad it gets.  It’s one of the worst things about computer hardware.

Edited July 7, 2022 by Bombastinator

[mariushm]

You probably want to say something other than UDP.

 

TCP and UDP are two different ways of transferring data between computers.

With TCP, data is packed in "data packets" which have headers that contain information like unique id for each packet, error correction codes and  is designed to be reliable, computer that receives data has to acknowledge a data packet was received, and then the computer on the other end can send more data, and the computer can also use the error correction information to verify the data packet is correct or request the data packet to be retransmitted if an error is detected.

 

UDP is sort of like a broadcast, it's a protocol used when you don't care if something arrives corrupted at the other end and there's no communication between computers (or more precisely with the network hardware) like with TCP - for example Youtube may use UDP to send a live stream to thousands of people, because if there's a small data packet corrupted somehow, the worst thing that could happen is for some green square to show up in one video frame in a video or at worst, you'd get one black frame in the live show, so nothing critical. It's also used by game servers and multiplayer games, because games send and receive the state of the "game world" lots of times each second, so if one update is corrupted somehow, it doesn't really matter, the next data packet will update the state and game and server will correct the player position, or just miss one bullet or some minor stuff like that.

 

FTP works over TCP and uses a permanent connection to send commands to the remote computer as if you'd type the commands in the command prompt ... list folder contents, retrieve file , resume next file i request from byte 1234, and other commands.

Each time you request a file through FTP, there's a second connection created and data gets moved through that second connection while the original connection remains active for commands.

This kind o behavior is not that cool, and it causes problems sometimes with firewalls

 

FTPS is encrypted FTP ... it's still FTP working on TCP protocol ... your FTP client connects to the remote ftp server and can ask the ftp server at the other end what features it supports and if the server says it supports FTPS extensions, your client can request to encrypt the communication from that point (or can connect directly to the ftp server on another port specially configured for ftps only connections)

 

Going a bit off topic...

 

HTTP is another protocol that works over TCP ... but unlike FTP, it's much simpler, you request a file or something from the server by sending some clear text with commands  (at least up to http 2, from http2 they added encryption and compression and other things) and then the server replies with the data your requested

For example, you could connect to a server with telnet or something (for example PuTTY) and type commands to get files directly sent to you, if the remote server supports it.

I started a super basic web server with PHP's built in server  

 

php -S localhost:9000 -t .

d:\Programs\php7>php -S localhost:9000 -t .
PHP 7.1.4 Development Server started at Thu Jul  7 21:26:07 2022
Listening on http://localhost:9000
Document root is D:\Programs\php7
Press Ctrl-C to quit.

 

This commands starts a web server on the local computer at address localhost and port 9000  (the default for http is 80)  and the web servers serves documents from "." which is the folder where php is located.

So now I can connect to localhost 9000 with putty (with the RAW connection, and check never close to see the reply) and type the http command to get a document

 

 

The simplest request is  in the form of 

 

GET [PAGE ADDRESS ]  HTTP/[VERSION]

an empty line tells the server you're done sending and the server replies with the data and closes connection, unless you tell it otherwise through the request

 

In the picture below, I had an index.html page in that folder and as I entered that empty line, the server replied with the document. That simple.

Server replies with some attributes for the reply like the first line says it's a http reply, the version the server understands, date of reply, it tells you that it will close the connection when it's done, and it tells you what kind of data it gives you back and the length of the data then it spits an empty line, followed by that content.