I've experienced this with assassin's creed unity and watch dogs. I've read several articles mentioning the Xbox has poor memory management resulting in memory leaks. The only way to deal with it is to do a cold boot; this purges the memory completely. Just let your game run for hours (i.e. more than 72 hours) and leave your console on stand-by. Did you experience your game freezing at some point?