First, square rooms are not so recommended for home theater use. but it's not an impossible task. A lot of people are using square rooms and are very happy with it. If you add a little room treatment, it will be perfect. Better to go with 2 subwoofers and move around rooms and find the ideal spot; this is the only difficult task. You can go with 7.2.4. ( later you can add power amp for that 2 additional channels )
AVR-denon 3800h
speakers: Qacoustics, Elac, JBL Studio (I don't know if it fits in your budget),monitor audio
subwoofer-svs,rel,ascendo( either of this will be a good options)
monolith,tone winner and rhythmic ( i dont know if its fit in your budget after avr and speaker purchase )