Best Service for...

[r0otctrl]

My current web-hosting provider does not allow root access. Hence I'm unable to install Scrapy, Selenium, and similar tools, to do web-scraping. My current web-hosting provider charges anywhere from $40 to upwards of $200 a month for VPS and dedicated hosting, which would allow me to do what I want to do, but I simply can't afford it.

All I'd like to do is be able to install Python, Docker, Splash, Selenium, et cetera, for scraping, and also be able to host websites to display the data.

Lots of people have suggested checking out AWS, and somebody else has suggested Kimsufi servers, but it has been difficult finding straightforward descriptions or explanations of exactly what each service offers.

Any help would be greatly appreciated!

[Mayaa]

What kind of specs do you need?

 

Im currently using DigitalOceans smallest VPS and its been pretty good so far.

[jj9987]

AWS, Scaleway, DigitalOcean, Google Cloud and many others - they all offer you a barebones virtual machine, you get root access and you can do whatever the heck you want with it (as long as you are within the limits of the ToS of the provider). And can be way cheaper too, depending on your hardware needs though.

[r0otctrl]

47 minutes ago, Mayaa said:

What kind of specs do you need?

 

Im currently using DigitalOceans smallest VPS and its been pretty good so far.

I don't need anything super heavy duty. I just need to be able to scrape the web continuously and host a website to display the data. I have zero experience when it comes to this stuff, so I'm not entirely sure what kind of specs I'd need for this!

What do you use for VPS for and how much are you paying per month?

[r0otctrl]

29 minutes ago, jj9987 said:

AWS, Scaleway, DigitalOcean, Google Cloud and many others - they all offer you a barebones virtual machine, you get root access and you can do whatever the heck you want with it (as long as you are within the limits of the ToS of the provider). And can be way cheaper too, depending on your hardware needs though.

Do any of their ToS prohibit web scraping?! I mean, until I get access to their API, one of the websites I wanted to scrape was Amazon. I imagine there's a good chance the AWS ToS prohibit this??

[jde3]

2 hours ago, jj9987 said:

AWS, Scaleway, DigitalOcean, Google Cloud and many others - they all offer you a barebones virtual machine, you get root access and you can do whatever the heck you want with it (as long as you are within the limits of the ToS of the provider). And can be way cheaper too, depending on your hardware needs though.

This.

Best answer this guy.

[r0otctrl]

13 minutes ago, jde3 said:

This.

Best answer this guy.

Which AWS service exactly? Also, have you heard of Kimsufi??

[jde3]

13 minutes ago, r0otctrl said:

Which AWS service exactly? Also, have you heard of Kimsufi??

Different plans are different amounts..

 

Digital Ocean will get you a single core with not much ram and not much space for $5 a month.. but.. its $5 a month.. hard to complain.

 

Kimsufi looks.... ok.. ish. I'd just go with DO myself for that price though because it's much more established.

( I thought it was related to Kim Dotcom, I was really excited.. nope. now I'm sad) DO charges by the hour so if you just want something to test for a few seconds you can do it and it's only a few cents. I also though Linus had a advertisement deal with them..?

[r0otctrl]

7 minutes ago, jde3 said:

Different plans are different amounts.

No, which service should I be looking at to do what I want to do?

I'll check out digitalocean again. It didn't seem like the best option when I scanned their website yesterday. Maybe I'm missing something.

[jde3]

2 minutes ago, r0otctrl said:

No, which service should I be looking at to do what I want to do?

I'll check out digitalocean again. It didn't seem like the best option when I scanned their website yesterday. Maybe I'm missing something.

You mean.. what.. type of platform you want? You choose that on setup so.. Ubuntu or FreeBSD or whatever. And they mail you the root SSH key.

[r0otctrl]

Just now, jde3 said:

You mean.. what.. type of platform you want? You choose that on setup so.. Ubuntu or FreeBSD or whatever. And they mail you the root SSH key.

No, of the AWS services, which service exactly would suit my needs?

[jde3]

3 minutes ago, r0otctrl said:

No, of the AWS services, which service exactly would suit my needs?

I've talked to you before on this and I still don't know what you want to do. your going to have to figure out the nuts and bolts yourself but... you can install anything on to DO you like. (if your in violation of ToS they will shut you down but..) You can even hack on other OS's on to it they don't provide like OpenBSD.. anything thats supported by KVM works.

 

If you go with AWS it's the same. Just more control really.

[r0otctrl]

Just now, jde3 said:

I've talked to you before on this and I still don't know what you want to do. your going to have to figure out the nuts and bolts yourself but... you can install anything on to DO you like. (if your in violation of ToS they will shut you down but..) You can even hack on other OS's on to it they don't provide like OpenBSD.. anything thats supported by KVM works.

All I need is to be able to install Python, some Python frameworks, a database (MySQL preferably), support for HTML, CSS, JS, PHP, and to host a website that will display the data?

[jde3]

1 minute ago, r0otctrl said:

All I need is to be able to install Python, some Python frameworks, a database (MySQL preferably), support for HTML, CSS, JS, PHP, and to host a website that will display the data?

Well.. your name is root control.. and they give you root so.. you got this right?

[r0otctrl]

1 minute ago, jde3 said:

Well.. your name is root control.. and they give you root so.. you got this right?

For which service?! There's like a thousand different services? I'm assuming you've used AWS before since you're recommending it?

[jde3]

15 minutes ago, r0otctrl said:

For which service?! There's like a thousand different services? I'm assuming you've used AWS before since you're recommending it?

How should I know. EC2 T2 general purpose?

 

And I'm recommending DO.

[r0otctrl]

8 minutes ago, jde3 said:

EC2 T2 general purpose.

 

And I'm recommending DO.

T2 as in tier 2 or?

[KirbyTech]

How good are you with Linux? If you are even half decent you could do Digital Ocean, they are great. I run everything from my personal backup site (if my home server goes down it spins up a DO instance) and a bunch of other services that I want to have available all the time like my status application. 

[r0otctrl]

13 minutes ago, KirbyTech said:

How good are you with Linux? If you are even half decent you could do Digital Ocean, they are great. I run everything from my personal backup site (if my home server goes down it spins up a DO instance) and a bunch of other services that I want to have available all the time like my status application. 

So I saw on the DO website that they support a LAMP setup. I know my way around, but I'm not great.

I'm looking at the bare-bones $5/month plan and I have a few questions for you:

1. Will the space of the operating system and other programs count towards that 25GB of SSD storage limit?

2. What is the clock speed of that 1vCPU (and that's a virtual CPU)?

3. 1GB of RAM?

I'm also looking at the storage pricing and the prices don't look reasonable at all. For instance, you can rent 4TB of SSD storage for $400/month -- sure, it's an SSD -- but you could buy a 4GB HDD for almost as little as a quarter of that price o:

[jde3]

2 hours ago, r0otctrl said:

So I saw on the DO website that they support a LAMP setup. I know my way around, but I'm not great.

I'm looking at the bare-bones $5/month plan and I have a few questions for you:

1. Will the space of the operating system and other programs count towards that 25GB of SSD storage limit?

2. What is the clock speed of that 1vCPU (and that's a virtual CPU)?

3. 1GB of RAM?

I'm also looking at the storage pricing and the prices don't look reasonable at all. For instance, you can rent 4TB of SSD storage for $400/month -- sure, it's an SSD -- but you could buy a 4GB HDD for almost as little as a quarter of that price o:

1. Yes, What they give you isn't very different from your own KVM drive or Virtual Box or VMWare. You can write whatever you want to the drive but it is only as big as it is. (You can buy block/object storage that is optional this drive is just for the OS and apps.)

 

2. freebsd@freefall # dmesg | grep CPU

CPU: Intel(R) Xeon(R) CPU E5-2650L v3 @ 1.80GHz (1797.97-MHz K8-class CPU)

typical..

 

3. Yes but, there isn't a desktop so... without chrome you don't need 128GB of ram. Servers take a lot less space.. granted this is pretty tight. the $5 ones are meant for micro services and testing mostly. You can convert them to what you need but there is added cost.

 

As for storage yes you could buy it. Go right ahead.. connecting that drive in your hand into your cloud infrastructure would be interesting to see. And 4TB of cloud storage is a HELL of a LOT of storage, what the hell are you doing anyhow?

[r0otctrl]

8 minutes ago, jde3 said:

As for storage yes you could buy it. Go right ahead.. connecting that drive in your hand into your cloud infrastructure would be interesting to see. And 4TB of cloud storage is a HELL of a LOT of storage, what the hell are you doing anyhow?

Oh, I wasn't implying connecting the 4TB to the cloud for storage on the cloud. What I was saying is that I don't see the point in paying $400 per month for 4TB of SSD storage when you can buy 4TB of HDD for a quarter of the price or even 2TB of SSD storage for almost the exact same price. However, if I were to buy a 4TB HDD, I'd just use it in my own server downstairs.

[jde3]

If you need something like that a NAS is more you speed yes. Amazon has better storage prices also (I'm pretty sure.)

 

As for ram to get an idea of what is possible I have 5 containers (bsd jails) running in one droplet each with various micro services in them. and it consumes 350~450MB of ram. so more than 50% free. (containers rule)

[r0otctrl]

1 hour ago, jde3 said:

If you need something like that a NAS is more you speed yes. Amazon has better storage prices also (I'm pretty sure.)

 

As for ram to get an idea of what is possible I have 5 containers (bsd jails) running in one droplet each with various micro services in them. and it consumes 350~450MB of ram. so more than 50% free. (containers rule)

What are containers? How come it consumes so little RAM?

I'm thinking of just using Heroku to get my product built and then perhaps upgrading the infrastructure afterwards when need be.

[jde3]

16 minutes ago, r0otctrl said:

What are containers? How come it consumes so little RAM?

I'm thinking of just using Heroku to get my product built and then perhaps upgrading the infrastructure afterwards when need be.

A container is a Virtual OS instance, kind of like a VM but more like a chroot. They behave like real systems, you can reboot them, log into them, do complex networking with them, mount drives in them etc etc. But unlike a VM they share the kernel image of the host and they don't over-provision like a VM. (they don't take more resources than they need because only one kernel controls the system resources)

 

The development of them was to solve the problem with the omnipotent root in Unix. (it wasn't to visualize apps) In the 80's and 90's this was a major criticism of Unix. So chroot and later Jailes evolved. As the application stack grew to the point where it consumed the entire platform, the industry moved more to containers VM's.  Why? mmm.. well most container systems only worked good on Sun and BSD is the easy explanation. It's hard work to implement this in an OS and a VM is much easier from a design standpoint. A VM also grants platform freedom, a container does not.

 

A few OS's have different ways to do this.. and those ways each have their own aspects and shortfalls. Linux has cgroups and namespaces (that is two things that provide container like abilities used together) FreeBSD has Jails. and Solaris/Illumos has Zones. There are then managers for those you can place on top like Docker or iocage.

[r0otctrl]

7 minutes ago, jde3 said:

A container is a Virtual OS instance, kind of like a VM but more like a chroot. They behave like real systems, you can reboot them, log into them, do complex networking with them, mount drives in them etc etc. But unlike a VM they share the kernel image of the host and they don't over-provision like a VM. (they don't take more resources than they need because only one kernel controls the system resources)

 

A few OS's have different ways to do this.. and those ways each have their own aspects and shortfalls. Linux has cgroups and namespaces (that is two things that provide container like abilities used together) FreeBSD has Jails. and Solaris/Illumos has Zones. There are then managers for those you can place on top like Docker or iocage.

Containers (or that type architecture if you'd call it that) sound pretty damn efficient. Much more than VMs.

I'm not sure which free service to go with. Either the Amazon free year of AWS or Heroku. You seem to get a lot more with the free year of AWS, though you can exceed it and end up paying more. On AWS I'd have to manage the server too, wouldn't i?