It's hard for me to tell from the photo what the brown algae is. If it is just encrusting, and does not 'erupt' from the surface I'd not be too worried about it.
MUCH MORE of a concern IMO/IME is that the hair algae you have might infact by Bryopsis hair algae, which can be a really big problem as few things eat it. Do the tips of the algae look feathery (like Bryopsis)? Does it pull off the rocks easily (like normal hair algae)?