Bell-Baggie will stay with the club until after the play-offs, should the Millers qualify.
McAllister will return to Exeter after the club's final regular season game against Hereford.
Ryan Taylor, currently on loan with the Grecians, will return to Don Valley at the same time.
Bell-Baggie told the club's official website: "I am now looking forward to the rest of the season and helping Rotherham to promotion.
"I want to try and grab this opportunity with both hands."
Source: BBC Sport
Source: BBC Sport