Best way to download an entire website?

[superbuu]

If you wanted to for instance download this entire forum, with every thread, post, image etc and be able to browse it online.. how do you do this properly so you get 1:1 copy with all the scripts and so on?

[badreg]

You don't, unless you have access to the servers and make a copy of the source and databases. Making a 1:1 copy of a website without server access is only possible if the website's code is exclusively HTML, CSS and JS.

 

The best you can do client-side is to scrape the content that you want, and reverse engineer the logic in order to display the content how it originally appears. But this is very bad practice, and something that is likely to get you banned by the target website.

 

Not everything on the web is archivable, as evidenced by the amount of broken sites that exist on archive.org.

[porina]

Software exists which will crawl a site and store a local copy of it as it looks like to a browser. However if anything is running on the back (e.g. a database), you'll wont be able to capture that, only the output. Like if you go and order a meal somewhere, you get the food, not the recipe for it.

 

Using a forum as the example, it would be impossible to get a working copy since it is dynamic and constantly changes. If you just wanted to index the posts, that could be possible as essentially it is what a search engine does. Sites may deploy protections against "unusual" activities like mass downloading, and the workaround I've used in the past is simply to not have more than 2 downloads running at a time, and spread it out over time. This may be impractical for obvious reasons if the site very large, unless you can parallel it up using VPNs for example so it doesn't look like the demand is from a single point.

[superbuu]

So which is the best way to go about this? what about this software?

https://iteroni.com/watch?v=9Fz7Xa-M8BQ

or

https://iteroni.com/watch?v=e6e5XDLaYRo

[mansouroussama]

https://www.httrack.com/

I used this and it worked okay, i would say it only works with static websites and it might take some time to fetch all website data (images, scripts..), it depends on the size of the website.

[FlappyBoobs]

+1 for httrack. I used it to scrape our companies internal wiki for audit. Takes a while to run but gets every public facing bit of data.