It could,probably, be done;but I have never heard of anyone doing this before. If you need the board to be small then a ITX one is pretty slim anyway, and you could just use extenders on things like USB.
I don't see why there would be performance loss if it was done.
The only thing is though is that most boards will have one port on top of another, but to be removed they will both have to be de-soldered, which would be hard to do as you might not be able to get a soldering iron in the gaps to heat it up. I would personally just use adapters or extenders, that way the board could be used in a different case later down the line.
It's a cool idea though.