The Castle once stood to the East of the present-day centre of the village. Its towers and battlements are known to have been in good repair in the twelfth century, when Henry II spent several days there in hunting and amusement. The Castle is thought to have lasted until the mid-fourteenth century, and in the nineteenth century a report states "the remains of Rowlands Castle consists of two masses of wall which are about 10 feet thick with a fosse of considerable depth".
The Parish of Rowlands Castle was formed in 1932 consisting of Blendworth Parish, Idsworth Parish (which had been attached to Chalton at one time) and parts of five other Parishes. One of these, Havant (Redhill), provided Rowlands Castle with a church, the Church of St. John, built about 1840 (registers date from 1841).
The 'Castle Inn' was in the possession of the Outen family for 200 years, and was said to have been the headquarters of a notorious gang of smugglers.
Idsworth House was erected in 1852, in the Elizabethan style, and was the property of Lorna, Countess Howe.
The ancient chapel of St. Hubert stands in the middle of a field, thought to be the site of a mediaeval village, on what was part of Idsworth Park. Parts of the chapel date from the twelfth century.