The Torroella family were the first to live in this fortified house. They lived under the rule of the counts of Empúries and were granted generous privileges in order to attract new settlers.

In the 13th century, the Palau was converted into the residence of the counts of Barcelona following Torroella de Montgrí's accession to the status of royal town in 1273. The subsequent years marked one of the Palau lo Mirador's main periods of prosperity.

As such, King James the Conqueror stayed here while on his way to the Council of Lyon. John I, Peter the Ceremonious' son and Martin the Humane's brother, also stayed at the Palau lo Mirador the night before he and his advisors were killed while hunting in the woods of Foixà in 1396, a mystery that has never been solved.

The second major period of prosperity began with the Marquis of Robert who, at the beginning of the 20th century, wanted to turn the building into a summer residence in the style of a Gothic palace. Rafael Masó, an architect of the Catalan Noucentisme movement, was commissioned to carry out the renovation project of the manor house, with the aim of making it a residence that was fit for nobility. He made use of different architectural elements from various other houses in Torroella belonging to the Robert family, such as stones, vaults, chapels, etc., in order to maintain the Medieval feel of the building.