Ceph vs OpenIO

[leadeater]

2 minutes ago, Eniqmatic said:

Indeed. ESX has some terribly infuriating behaviour sometimes but I had to start using it at home because Hyper-V and certain Linux distributions that I wanted/need to use had terrible support from Microsoft (big surprise) and their documentation of the LIS was terrible. That was about a year or so ago though, might have changed.

 

Your use case and performance expectations and therefor your hardware are much higher than mine so I don't think I would really be able to provide you with relevant stats on Gluster unfortunately. Setup is extremely easy however I will comment on that.

Well the fact that it is easy to setup is more than helpful, OpenIO's claim of the simplest to setup is coming up well short. By now for the same time spent I'd be able to actually write data in to Ceph.

[Eniqmatic]

4 minutes ago, leadeater said:

Well the fact that it is easy to setup is more than helpful, OpenIO's claim of the simplest to setup is coming up well short. By now for the same time spent I'd be able to actually write data in to Ceph.

With Gluster you can literally take 2 VM's and have everything installed and working between them in ~5 minutes. It is seriously quick. I have done literally 0 work on looking at any tuning options so far but in terms of getting from nothing to seeing something happening, very quick.

 

If you have a spare 2 machines you could try it on and run the same benchmarks I'd be interested to see what performance you get.

[leadeater]

On 9/24/2016 at 2:57 AM, Eniqmatic said:

With Gluster you can literally take 2 VM's and have everything installed and working between them in ~5 minutes. It is seriously quick. I have done literally 0 work on looking at any tuning options so far but in terms of getting from nothing to seeing something happening, very quick.

 

If you have a spare 2 machines you could try it on and run the same benchmarks I'd be interested to see what performance you get.

I setup Gluster, even easier than Ceph .

 

Performance is really good, CPU usage is also high though. Just 3 nodes with a single SSD each makes the host hit 80% CPU going ~1GB/s. Rather annoying since I don't actually need the speed, slower SSDs would be perfect lol.

 

I am having an issue with it though, I suspect CPU related. When I push the system hard often the whole gluster volume locks up and stops responding and doesn't self recover without a reboot of the server (could just do services but eh reboot is quick). The gluster client that is mounting the volume is also a host node and when I run performance tests that VM hits 100% CPU.

 

I'm actually liking Gluster more than Ceph for what I want but I need it to be stable.

[Eniqmatic]

6 hours ago, leadeater said:

I setup Gluster, even easier than Ceph .

 

Performance is really good, CPU usage is also high though. Just 3 nodes with a single SSD each makes the host hit 80% CPU going ~1GB/s. Rather annoying since I don't actually need the speed, slower SSDs would be perfect lol.

 

I am having an issue with it though, I suspect CPU related. When I push the system hard often the whole gluster volume locks up and stops responding and doesn't self recover without a reboot of the server (could just do services but eh reboot is quick). The gluster client that is mounting the volume is also a host node and when I run performance tests that VM hits 100% CPU.

 

I'm actually liking Gluster more than Ceph for what I want but I need it to be stable.

I also got past my Gluster issues last night/today and started to test more with it.

 

I too hit some CPU issues today but in an odd manner. But I don't have near you're hardware so I am probably being bottle-necked on other hardware before I am seeing high CPU.

 

I haven't had that issue at all with locking up, but once again probably due to the above.

 

My issues today were that I went live on two web servers today, in a 4 node cluster. 2 of the servers are web servers and 2 are just backup storage servers (so they contain all the same files just not the web services to run them if that makes sense). The 2 web servers became really slow and unresponsive as soon as I moved all the data onto the Gluster Volumes. Even though there wasn't much data. They became so slow that the load balancer health checks were failing on them. Only thing I can think of was that access logs were stored on the volume and the load balancer health checks would be logged there, thus changing the file every second on each server. I don't know, need to investigate.

 

So my next plan is to either reduce the node count to 2 or 3, to see if this helps, or do some tuning with the options of which I have not done any yet.

 

Your problems I think could be resolved using the tuning options. Have you looked in the volume log when it happens? Are you storing the volume on its own disk?

[leadeater]

5 hours ago, Eniqmatic said:

I also got past my Gluster issues last night/today and started to test more with it.

 

I too hit some CPU issues today but in an odd manner. But I don't have near you're hardware so I am probably being bottle-necked on other hardware before I am seeing high CPU.

 

I haven't had that issue at all with locking up, but once again probably due to the above.

 

My issues today were that I went live on two web servers today, in a 4 node cluster. 2 of the servers are web servers and 2 are just backup storage servers (so they contain all the same files just not the web services to run them if that makes sense). The 2 web servers became really slow and unresponsive as soon as I moved all the data onto the Gluster Volumes. Even though there wasn't much data. They became so slow that the load balancer health checks were failing on them. Only thing I can think of was that access logs were stored on the volume and the load balancer health checks would be logged there, thus changing the file every second on each server. I don't know, need to investigate.

 

So my next plan is to either reduce the node count to 2 or 3, to see if this helps, or do some tuning with the options of which I have not done any yet.

 

Your problems I think could be resolved using the tuning options. Have you looked in the volume log when it happens? Are you storing the volume on its own disk?

Looked at every log I could find, can't find any reason for it. I'm going to use a dedicated glusterfs client next to mount the volumes and serve NFS/iSCSI rather than cheating for time sake and using one of the storage nodes. 

[leadeater]

@Eniqmatic Moving the gluster mount to a dedicated client fixed the locking up issue for replicated volumes.

 

I do have one issue still though, original reason I changed to CentOS from Ubuntu, for striped volumes I'll be writing a test file with dd and it will lock up the client/session. If I ssh in to the same gluster client with a new session the mount is still working and I can create a text file just fine, I can even delete the file that dd was writing to. Really weird and I can only guess it has something to do with write coherency or something, it shouldn't be an issue though I mean it is a valid volume type to pick from.

 

Reason I moved to CentOS was most guides used that or fedora for Ceph/Gluster, makes sense though since both are now owned/created by red hat.

 

Right now as it looks Gluster is easier to setup but is having some really odd behavior and Ceph while more complex configuration wise and resource demanding just works without issue.

[unijab]

I had looked at gluster before, and may do more tests with it, but I like the idea that ceph can do block level storage. So I may change around my storage server to do some testing with ceph.

I'd also love to build a DIY 10Gb ethernet switch sometime also.

 

 

[leadeater]

6 hours ago, unijab said:

I had looked at gluster before, and may do more tests with it, but I like the idea that ceph can do block level storage. So I may change around my storage server to do some testing with ceph.

I'd also love to build a DIY 10Gb ethernet switch sometime also.

Yea really like Ceph, pitty I'm having so much issues with my SSDs though. I did however find out this is a known issue with 3 or 4 brands for Linux and TRIM/Discard, Samsung and Micro being some of the ones I can remember.

[Eniqmatic]

On 25/09/2016 at 7:43 AM, leadeater said:

@Eniqmatic Moving the gluster mount to a dedicated client fixed the locking up issue for replicated volumes.

 

I do have one issue still though, original reason I changed to CentOS from Ubuntu, for striped volumes I'll be writing a test file with dd and it will lock up the client/session. If I ssh in to the same gluster client with a new session the mount is still working and I can create a text file just fine, I can even delete the file that dd was writing to. Really weird and I can only guess it has something to do with write coherency or something, it shouldn't be an issue though I mean it is a valid volume type to pick from.

 

Reason I moved to CentOS was most guides used that or fedora for Ceph/Gluster, makes sense though since both are now owned/created by red hat.

 

Right now as it looks Gluster is easier to setup but is having some really odd behavior and Ceph while more complex configuration wise and resource demanding just works without issue.

Sorry, just to clarify, you are now not having the issue when using a separate machine for the client and using the replication structure, but you are when you change to a distributed structure? That's odd. I don't know if you've used/use IRC at all (I personally have actually never touched it, I know right) but apparently the Gluster IRC is very helpful.

[leadeater]

16 minutes ago, Eniqmatic said:

Sorry, just to clarify, you are now not having the issue when using a separate machine for the client and using the replication structure, but you are when you change to a distributed structure? That's odd. I don't know if you've used/use IRC at all (I personally have actually never touched it, I know right) but apparently the Gluster IRC is very helpful.

When using replication between bricks everything is fine, after I moved the gluster client to a dedicated server. When I use striped bricks I get the locking up issue, kinda weird.

 

I've deleted all my gluster servers now though since it's not quite as close to what I want as Ceph is.

 

I have learn't however when researching in to SSD linux optimization that Samsung firmware is a bit buggy and reports false features which screws with things like TRIM/Discard.

Quote

In particular, many drives, including Samsung, Micron, Crucial have problems with discard/TRIM. Also see 790520

https://wiki.debian.org/SSDOptimization

 

Quote

It was recently discovered that newer firmware versions for Samsung 8* SSDs 
claim to support SATA 3.2 features, but that NCQ TRIM has data corruption 
bugs. For this reason that feature was added to the blacklist with this 
commi

 

Quote

/* devices that don't properly handle queued TRIM commands */
	{ "Micron_M500_*",		NULL,	ATA_HORKAGE_NO_NCQ_TRIM |
						ATA_HORKAGE_ZERO_AFTER_TRIM, },
	{ "Crucial_CT*M500*",		NULL,	ATA_HORKAGE_NO_NCQ_TRIM |
						ATA_HORKAGE_ZERO_AFTER_TRIM, },
	{ "Micron_M5[15]0_*",		"MU01",	ATA_HORKAGE_NO_NCQ_TRIM |
						ATA_HORKAGE_ZERO_AFTER_TRIM, },
	{ "Crucial_CT*M550*",		"MU01",	ATA_HORKAGE_NO_NCQ_TRIM |
						ATA_HORKAGE_ZERO_AFTER_TRIM, },
	{ "Crucial_CT*MX100*",		"MU01",	ATA_HORKAGE_NO_NCQ_TRIM |
						ATA_HORKAGE_ZERO_AFTER_TRIM, },
	{ "Samsung SSD 8*",		NULL,	ATA_HORKAGE_NO_NCQ_TRIM |
						ATA_HORKAGE_ZERO_AFTER_TRIM, },
	{ "FCCT*M500*",			NULL,	ATA_HORKAGE_NO_NCQ_TRIM |
						ATA_HORKAGE_ZERO_AFTER_TRIM, },

 

Going to move the hypervisor to Windows Server 2016 since I know the SSDs are totally fine on that OS and create VHD's on each SSD for for the Ceph nodes. Hopefully this will allow me to get round the issue and properly test it. I'll also give Gluster another shoot if it does work.

[Eniqmatic]

23 hours ago, leadeater said:

When using replication between bricks everything is fine, after I moved the gluster client to a dedicated server. When I use striped bricks I get the locking up issue, kinda weird.

 

I've deleted all my gluster servers now though since it's not quite as close to what I want as Ceph is.

 

I have learn't however when researching in to SSD linux optimization that Samsung firmware is a bit buggy and reports false features which screws with things like TRIM/Discard.

https://wiki.debian.org/SSDOptimization

 

 

 

Going to move the hypervisor to Windows Server 2016 since I know the SSDs are totally fine on that OS and create VHD's on each SSD for for the Ceph nodes. Hopefully this will allow me to get round the issue and properly test it. I'll also give Gluster another shoot if it does work.

I think I will try Ceph today. I'm barely making any changes to files but as I say as soon as I start allowing traffic to a Gluster volumes where two servers are changing files at the same time, performance tanks. Its a shame because its so easy to setup.

 

How is the Ceph setup and reliability/stability? Looks to be good from what you've said so far?