Patches are necessarily derivative code, so covered by the license.

However the GPL never said you have to make your code freely available to the public. It's just that once you give it out, you can't stop anyone else from redistributing it, if they want to.