Probably depends on the model of bike you have. Since you posted in the touring forum section, then I suppose that's what you're riding. Not bashing ya, just helps you get good info if you let everybody know what ya got. I didn't have to remove any brake hardware when I swapped out the exhaust on my wife's 2012 Street Glide or my 2010 Wide Glide. You WILL have to remove the right side rear foot-board bracket bolt (driver's foot-board), loosen the front bolt, and swing it down out of the way to get the stock exhaust off and install the new exhaust (or remove the foot-board completely). Ask me how I know. I'm hardheaded and don't like taking unnecessary steps if I can help it, but sometimes it comes around and bites me in the @ss.
Anyway, good luck with the removal and install.