The museum is situated in a 17th century Grade I listed house, which is regarded as one of the most important buildings in Warwick. The mediaeval Hospital of Saint John the Baptist was founded by the Earl of Warwick on the site around the year 1154. On the Dissolution of the Monasteries the land was passed to the Stoughton family. Between 1666 and 1670 Nathaniel Stoughton rebuilt the house very much as it is today. The two-storey house with attics for the servants has five distinctive bays; three ‘Dutch ’ and two triangular gables. It is made from ‘Warwick stone ’, the local sandstone. When it was new the house, with its grounds and gates, would have been very imposing, reflecting the importance of Nathaniel Stoughton. In 1791 it became a private school and later, during the 20th century, was taken into use as a military record office; it has been a museum since 1961.