After the conquest of the city by the Portuguese troops, Afonso de Albuquerque had the fortress restored and the first captain of the city, Rodrigo Rebelo, inhabited it.
In 1554, Viceroy Pedro de Mascarenhas settled there, abandoning the Palace of Sabaio, which had been the residence of the viceroys in the first half of the sixteenth
century. The site, near the banks of the Mandovi, overlooking the estuary, was closer to the situation of the Royal Palace of Ribeira, which influenced the permanence of
the viceroys in the Palace of Fortaleza and the development of this residence in an official palace. Architecturally, the palace was characterized by a set of different
evolutionary phases that succeeded each other organically, integrating the Trunk Chain and Royal Arms, which served as a storage and ammunition. The building acquired a
certain sumptuousness, referred to by several travelers, more by scale and proportions than by an elaborate architectural design. The palace survey, carried out in 1779,
bears witness to this. From the drawing there is a tendency for the chandelier architecture, and a sense of organicity accentuated by a set of high scissor ceilings that
simultaneously gave it a marked monumentality. In its distributive design, the palace was built on a walled courtyard, the entrance marked by a vast porch with two flights
of stairs covered with four-sided roof resting on thick stone columns. On either side of the porch were opened, in turn, the whole width of the elevation, two wide porches
of columns, a solution that would later be adopted in the great houses of the eighteenth century. As indicated by Pyrard de la Valle, this courtyard assumed important
functions of representation, where the whole aristocracy was assembled, on horseback or in machilas, during official ceremonies or when the Viceroy moved outside the
palace. As a paradigm of royal palace, the building added a vast chapel with tribune, whose connection with the interior of the palace endowed with characteristics of
palatine chapel. In its interior complex, the palace was endowed with two large rooms, a first one with an antechamber function, where the viceroy's personal guard was
kept daily, and a second reserved for council and large receptions. In the first room were the paintings of all the armadas. This room gave access to an even larger one,
decorated in its turn with the full-length portraits of all the viceroys, a precious collection that is now in the Franciscan convent of the city of Goa. In 1695, due to
the epidemic that devastated the city, the Count of Vila Verde was obliged to abandon it and to settle in the Palace of the House of the Gunpowder, in Panelim, outside Goa.
Already in the nineteenth century, the vice kings settled in Panjim, following the great works carried out in this locality and their passage to the capital of the State of
India. Even after the Palace of the Fortress ceased to be the residence of the viceroys, for many years the great courtroom was still used as a place of official receptions.