The subframe mounting points are different in the 03/04 to the 05/06. On the 03/04, the clamp pinched together at the top and bolted to the subframe above the can.
On the 05/06, the exhaust bolts to both sides of the subframe.
The other point where you will have a problem will be the mounting point underneath the bike. They are different, 03/04 to 05/06.
All in all, you can make it work, but you have to be open to fabricating brackets, etc.