Open source distributed “twitter”

[alexanderc]

Hi all,

 

Just watching the wan show and what Luke was saying about how easy it is to create a twitter clone got me wondering how viable would it be to run a distributed self hosted-ish twitter clone. 

The issue that I see with something like mastodon is that you still need to host it, you could either host it in the cloud which would incur monthly costs or build your own server with just an upfront cost, but if you go with hosting it on your own hardware, you would potentially have outages that are out of your control. 

 

Now, what if you would use a distributed shard based document db like elasticsearch to build such a clone, where each server would be in control of the primary shards but all replicas would be distributed across other users servers which would take over as primaries in case of an outage on the main users server. This could distribute the cost of such a platform across all the users, minimizing the revenue that it would need to make, thus reducing the cost required to run it, which would avoid the need of collecting user data in order to generate ad based revenue.

 

I’m also thinking that the documents that each server would generate would be fairly low, thus allowing the replication of data to happen somewhat ok over a 1Gb isp provided uplink as long as it’s in the same region and you could limit the number of replicas that one server could handle to not stress the available bandwidth.
 

The platform itself would be in control of the main nodes that would cluster all the data nodes, as well as the main web ui that would send queries on the relevant data nodes as well as handle data ingestion.

 

I’m thinking that you could assign a unique Id to each server that would match the data stream created in elasticsearch so when leaving a comment or creating a post the platform would create a document on that data stream. All of the generated indices (coming out of the data stream) would have a common pattern that would allow searches or document retrieval to happen across all of them and considering that elasticsearch is good for searching that would be fairly quick even if highly distributed.

Due to the distributed nature of it you wouldn’t need expensive hardware either, I think you could even repurpose older usff machines that can be bought cheaply and don’t take a lot of space in someone’s home.

 

Programmers, how viable would something like this be?

 

I would have posted this on Reddit but that’s messy right now as well…

[NocTheRocc]

Software engineer here, this looks like a solution in search of a problem.

 

Yes, building a twitter-like system using distributed computing is possible, but it's going to either fail to scale to a large number of users, have horrific latency, or fail to show all users the same feed. At the end of the day the CAP theorem is going to eat your lunch.

 

There's a reason most services aren't decentralized, because there are huge costs if you don't centralize to a single source of truth. Those costs are only worth it specifically if there's a very good reason that a central server can't be trusted, which just isn't true for a twitter-like service. Yes this means that someone is going to have to pay for server costs, but if you really need it just use the AWS free tier that gives you one of their really basic servers for free? What exactly are the benefits you're imagining for a system like this that mastodon doesn't have?

 

[Sauron]

2 hours ago, NocTheRocc said:

Software engineer here, this looks like a solution in search of a problem.

 

Yes, building a twitter-like system using distributed computing is possible, but it's going to either fail to scale to a large number of users, have horrific latency, or fail to show all users the same feed. At the end of the day the CAP theorem is going to eat your lunch.

 

There's a reason most services aren't decentralized, because there are huge costs if you don't centralize to a single source of truth. Those costs are only worth it specifically if there's a very good reason that a central server can't be trusted, which just isn't true for a twitter-like service. Yes this means that someone is going to have to pay for server costs, but if you really need it just use the AWS free tier that gives you one of their really basic servers for free? What exactly are the benefits you're imagining for a system like this that mastodon doesn't have?

To add to this, if the problem is money and enough people are interested that you'd consider going decentralized you could just crowdfund instead.

[10leej]

You literally described Mastodon, which you don't have to self host and never HAD to self host, you can join someone else's instance.

[Raytsou]

1 hour ago, 10leej said:

You literally described Mastodon, which you don't have to self host and never HAD to self host, you can join someone else's instance.

I was thinking it sounds like he was describing fediverse, but I hear it's closer to a decentralized reddit. I'm not too familiar with the technical implementation but I suspect a decentralized twitter might actually not be possible.

[HarryNyquist]

On 7/8/2023 at 1:55 PM, Raytsou said:

I was thinking it sounds like he was describing fediverse, but I hear it's closer to a decentralized reddit. I'm not too familiar with the technical implementation but I suspect a decentralized twitter might actually not be possible.

Fediverse is DEFINITELY not like decentralized reddit, lmao. It's absolutely decentralized twitter.