File Transfer through Ethernet Crossover

[metaleggman]

Hi all,

 

I recently built a new rig, and decided to throw some files from my laptop to said rig to free up disk space on the laptop.  I've been able to share files through my existing network fairly easily (for things like my Bluesound Node [though it hates my laptop for some reason; must be a random setting or something], moderate file transfer, Plex, etc.) by setting up shared folders via my wireless network.  However, and this is something I've been finding difficult for over a decade now, it seems like there's really no simple way to connect computers via an Ethernet cord within Windows (in the past, a crossover cable, but since all my devices seem to have gigabit now, thankfully I haven't had to dig out my old crossover cable from God knows where lol).  I needed to set this up because, for hundreds of thousands of files, sized around 120gb total, it was taking forever over Wi-Fi.  Now, I was able to get it working fairly easily, since I know how to assign manual ip addresses in the network adapter properties and what not, but, like, that seems like a lot of work to do for a one time deal, especially since I'm going to make sure to go back and reverse my steps just in case said settings mess up something in the future.  Is there a dead simple way to network two computers together in this way?  Something along the lines of, plug in the cable, open a networking wizard, and then choose what folders to share (either within the wizard or manually in the folder's properties).  Frankly, as it stands, it seems like it would be simpler to just setup a bittorrent file, or get a big enough usb 3 thumb drive lol.

 

Any thoughts are welcome.  Obviously if I had a proper wired network setup, this wouldn't really be an issue.  Alas, I do not.  Am I crazy, or lazy, (from a security stand point) to even want this?  I remember even as a kid, still on dial-up, having to resort to floppies to transfer files between computers, since we didn't have cd-burners or zip drives, and always wondering why I couldn't just easily take a cable between the two and start transferring files!

[piratemonkey]

8 minutes ago, metaleggman said:

Hi all,

 

I recently built a new rig, and decided to throw some files from my laptop to said rig to free up disk space on the laptop.  I've been able to share files through my existing network fairly easily (for things like my Bluesound Node [though it hates my laptop for some reason; must be a random setting or something], moderate file transfer, Plex, etc.) by setting up shared folders via my wireless network.  However, and this is something I've been finding difficult for over a decade now, it seems like there's really no simple way to connect computers via an Ethernet cord within Windows (in the past, a crossover cable, but since all my devices seem to have gigabit now, thankfully I haven't had to dig out my old crossover cable from God knows where lol).  I needed to set this up because, for hundreds of thousands of files, sized around 120gb total, it was taking forever over Wi-Fi.  Now, I was able to get it working fairly easily, since I know how to assign manual ip addresses in the network adapter properties and what not, but, like, that seems like a lot of work to do for a one time deal, especially since I'm going to make sure to go back and reverse my steps just in case said settings mess up something in the future.  Is there a dead simple way to network two computers together in this way?  Something along the lines of, plug in the cable, open a networking wizard, and then choose what folders to share (either within the wizard or manually in the folder's properties).  Frankly, as it stands, it seems like it would be simpler to just setup a bittorrent file, or get a big enough usb 3 thumb drive lol.

 

Any thoughts are welcome.  Obviously if I had a proper wired network setup, this wouldn't really be an issue.  Alas, I do not.  Am I crazy, or lazy, (from a security stand point) to even want this?  I remember even as a kid, still on dial-up, having to resort to floppies to transfer files between computers, since we didn't have cd-burners or zip drives, and always wondering why I couldn't just easily take a cable between the two and start transferring files!

have you tried to use ethernet on laptop, and just make both computers available on the network? i think you would be able to use file explorer and just drag and drop then. im not sure, as im not baller enough to have 2 computers lol

[mariushm]

You don't need a crossover cable.

All Gigabit network cards should support auto pair detection, they detect that another network card is at the other end of the cable and figure out through which pairs of wires they're supposed to send or receive data.

Crossover cables are only needed with 100 mbps ethernet cards.

 

So basically the fastest way... connect a regular ethernet cable between computers, or connect the two computers to a network switch.

Give each computer an unique IP address. For example, give the first computer the IP 192.168.0.1  and the second IP 192.168.0.2   , and to each computer give the same subnet mask 255.255.255.0

This tells each network card that they're in the same "network" and they get an IP without requesting one from a router or some switch or device on the network (which there's none, if you connect the computers together with a simple cable, there's just two computers in the network)

Now, you have a network between the two computers and you can transfer files. Easiest would probably be to share a folder on the network, and from the other computer you can simply browse in Windows Explorer and see that folder shared.

 

What I prefer to do is to install Filezilla FTP Server on one of the computers and Filezilla FTP Client on the other computer - doesn't matter which one because you either use Filezilla FTP Client to download the files from one computer to the other, or you can use Filezilla FTP Client to upload the files from one computer to the other.

 

Filezilla FTP Server takes less than 5 minutes to install and configure, basically install and create a ftp user with its password and set a folder to be served when someone connects using that username and password.

Filezilla FTP Client is equally easy to install and run, there's just one or two options to set and you're good to go (set all transfers to binary and the number of simultaneous connections)

 

Here's a tutorial I made for someone maybe a couple years ago : http://savedonthe.net/a/tutorial_filezilla.mp4

There's no audio, because it was in Romanian and wouldn't help you, but it's easy to follow and you can send me a message if something isn't clear.

 

The benefit of using FTP is that you can transfer multiple files in parallel, and there's resume support, in case the connection breaks or one computer has to restart or something.... parallel transfers help speed up transfers significantly when dealing with lots of small files.. with network shares, you're always transferring just one file at a time.

 

[metaleggman]

1 hour ago, mariushm said:

You don't need a crossover cable.

All Gigabit network cards should support auto pair detection, they detect that another network card is at the other end of the cable and figure out through which pairs of wires they're supposed to send or receive data.

Crossover cables are only needed with 100 mbps ethernet cards.

I'm aware, but thank you for clarifying for others.

2 hours ago, metaleggman said:

via an Ethernet cord within Windows (in the past, a crossover cable, but since all my devices seem to have gigabit now, thankfully I haven't had to dig out my old crossover cable from God knows where lol)

 

1 hour ago, mariushm said:

So basically the fastest way... connect a regular ethernet cable between computers, or connect the two computers to a network switch.

Give each computer an unique IP address. For example, give the first computer the IP 192.168.0.1  and the second IP 192.168.0.2   , and to each computer give the same subnet mask 255.255.255.0

This tells each network card that they're in the same "network" and they get an IP without requesting one from a router or some switch or device on the network (which there's none, if you connect the computers together with a simple cable, there's just two computers in the network)

Now, you have a network between the two computers and you can transfer files. Easiest would probably be to share a folder on the network, and from the other computer you can simply browse in Windows Explorer and see that folder shared.

Yeah, that's what I resorted to.  My question was more if there was a simpler way to do this within windows itself, like through a networking wizard or something.

 

1 hour ago, mariushm said:

What I prefer to do is to install Filezilla FTP Server on one of the computers and Filezilla FTP Client on the other computer - doesn't matter which one because you either use Filezilla FTP Client to download the files from one computer to the other, or you can use Filezilla FTP Client to upload the files from one computer to the other.

 

Filezilla FTP Server takes less than 5 minutes to install and configure, basically install and create a ftp user with its password and set a folder to be served when someone connects using that username and password.

Filezilla FTP Client is equally easy to install and run, there's just one or two options to set and you're good to go (set all transfers to binary and the number of simultaneous connections)

Oooh I forgot about FTP.  I'll give that a shot next time.  Currently I'm about a third of the way through my file transfer (started around the time I posted this lol).

Edited March 19, 2020 by metaleggman
To seem less snarky lol

[beersykins]

4 hours ago, metaleggman said:

hundreds of thousands of files

That likely won't speed up your transfer, then.  A huge amount of small files adds a lot of protocol latency and overhead down to the storage layer.

 

Otherwise, technically you could just connect them together and communicate with the APIPA address they assign themselves.  It's more convenient to just throw a static address in a different subnet for that ptp link though.

[metaleggman]

22 minutes ago, beersykins said:

That likely won't speed up your transfer, then.  A huge amount of small files adds a lot of protocol latency and overhead down to the storage layer.

 

Otherwise, technically you could just connect them together and communicate with the APIPA address they assign themselves.  It's more convenient to just throw a static address in a different subnet for that ptp link though.

TBF, it's a mix of lots of small files, and some very large files (thankfully in separate directories, HD videos mostly); It definitely helped with the large files.  I did notice a bit of a speed bump with the small files, but mainly in just Windows' collation or whatever before the transfer began.  Didn't think to try the APIPA addresses tho.  I think using something like ftp in the future, like mariushm mentioned will probably be the easiest solution, since I can at least pause and start it (case in point, I didn't have >260 character filepaths enabled, and had to go through folders to see what did and didn't transfer...)

Looks like the answer to my question is, more or less, a big fat no lol.  Womp womp.  Thanks for the help people.

[mariushm]

59 minutes ago, metaleggman said:

TBF, it's a mix of lots of small files, and some very large files (thankfully in separate directories, HD videos mostly); It definitely helped with the large files.  I did notice a bit of a speed bump with the small files, but mainly in just Windows' collation or whatever before the transfer began.  Didn't think to try the APIPA addresses tho.  I think using something like ftp in the future, like mariushm mentioned will probably be the easiest solution, since I can at least pause and start it (case in point, I didn't have >260 character filepaths enabled, and had to go through folders to see what did and didn't transfer...)

Looks like the answer to my question is, more or less, a big fat no lol.  Womp womp.  Thanks for the help people.

The maximum number of connections in Filezilla ftp client is 10, but there's no rule saying you can't open multiple Filezilla ftp clients.

You just have to be careful so that each ftp client downloads different folders ... for example have one client download C:\Games, another download C:\Movies  and so on

Another trick is using one client for one partition (ex c:\) and another client for another partition (ex d:\ )

 

Another trick i sometimes use is to configure one download thread, drag a big folder over and wait until all files are added to transfer queue. Then, you can right click on the queue and pause transfers, then sort the queue by file sizes and select all BIG files and raise the priority to those big files. Now Filezilla will transfer the big files, and leave the small size files at the end.

You can now limit the number of parallel connections to 3-5 if the big transfers trash your mechanical drive (reading 3-5 4GB+ files in parallel should easily saturate your network card, and do 125 MB/s)  and when those big transfers are done, you can raise the parallel connections to the maximum configurable of 10 and finish with the small files.

 

With small files, multiple parallel connections helps as while a connection is closed and new open is opened for a new file transfer, you have other 9 connections active, so at any point you're actually transferring something, so overall you get a higher transfer speed.

Mechanical drives won't have any issues with getting 10 random files at a time, as often those small files are within proximity of the other, so most likely they're gonna be in those 32-64-128 MB of cache on the mechanical drive, there's not gonna be so much seeking to cause the drive to slow down.

With big files, mechanical drives can sustain 125 MB/s even when transferring 10 big files in parallel ....

SSDs have no problems in all scenarios ... they can keep up with the gigabit of the network card.

 

[beersykins]

1 hour ago, metaleggman said:

Looks like the answer to my question is, more or less, a big fat no lol.  Womp womp.  Thanks for the help people.

Just hook everything to a switch/your-router bro and have a real network