The downside to a brake is absolutely the noise. I have a .338 WinMag with a brake on it.

All you need is one episode of buck fever, and you light that stupid thing off without hearing protection screwed into your ears, and that's it. the first round you shoot from a rifle with a chamber pressure over 60,000 PSI will cause you hearing damage. Somewhere, in all my digital crap, I have a plot of my hearing from an audiologist. You can see the "notch" in the frequency response in my right ear that is the result of shooting one round without hearing protection from a .338 with a brake, which is a cartridge that's running about 63,000 PSI.

Suppressors, if allowed in your state, will give much more recoil reduction, and they will bring the report of these high-pressure magnums down into a range where mere earplugs will protect your hearing.

If you don't want a silencer, there are other ways to reduce recoil. Mercury recoil reducers help, and can be installed in the buttstock of a rifle, for example.