Yes, the fan sound will be irritating if you are especially using an older one. I initially had only fan in my HT room, went mad and then got an AC installed. Though i dont feel much disturbance while listening to music, it does affect the performance while watching movies.
I had once attached my PC to LCD in main room as temporary HTPC. Its fan used to make a loud noise. I also have 5.1 HT speakers. Its true that in high volume scenes (like war) you rarely notice the fan sound. However, if your attention gets drawn towards it, it becomes irritating & spoils everything particularly if you are listening to music...