I know I'm late to the party and you probably already have a solution, but I'm surprised no one mentioned nas4free
I believe it's based on earlier version of freeNAS, but I'm not sure.
At any rate, I've had great success
I tried freeNAS originally and it just was too resource intensive. I was using a 3ghz Core Duo (not even a core2duo - this is an old one) with 6gb ram. I was able to run Plex and transmission - but it didn't perform well. Direct file transfers weren't bad, but it couldn't transcode at all.
Then I found nas4free
With Plex and Transmission running (but nothing actively transcoding) it's showing 0% cpu. freeNAS did't do that. Right now it shows I'm at 84% of my 6gb ram, but I've seen it lower.
I am able to transcode most 1080p videos on the fly without stutter, and with pretty quick startup.
http://www.nas4free.org/
I actually came here because I was trying to find a comparison of Rockstor to nas4free, so I couldn't tell you which will work better
Did you ever get your system going? What did you choose?