The castle (manor house) of Petar Zrinski, dating back to 1651, commands the Čabar town centre. It is made up of four buildings encircling a central court and two towers interconnected through a fortification wall. Rumour has it that there is a secret tunnel connecting the ground floor of the castle with the Tropetarska Stijena peak atop of Čabar, providing an escape route in the event of danger. No record of such a tunnel exists, apart from testimonies of the Čabar elderly who, before the entrance was walled off for safety, used to find old weapons and human bones in the tunnel.

To this day, a part of the original walls, tower and vaults on the ground floor of the central building have been preserved. The castle houses a historical and ethnological collection, a gallery of artwork by Vilim Svečnjak and a Routes of the Frankopans Interpretation Centre.