The Castle was built in 1767, during the reign of Carlos III.  It gets its name from the Duquesa de Arcos, wife of D. Luís Cristóbal Ponce de León, master of the lands in Casares and keeper of the peace of this area in 1569.

The castle was built for a number of reasons: to defend of the coast, customs control, protection against seaborne landings and as a base for infantry, cavalry and artillery forces.

The fortress was built on existing foundations laid down in Roman times with sloping walls on the outside of which there are loopholes for riflemen.  All internal areas were arranged around a central courtyard covered with practical terraces and rooms.

The wall facing the sea had an artillery platform, with a battery of 4 x 24 pounder cannons (able to fire 24 pound cannon balls). This platform was separated from the rest of the building by a ditch meaning it could be defended even if the main building was overrun.

The castle's garrison consisted of 25 infantry, 10 cavalry and 10 artillerymen plus their respective officers. During the early years of the 19th century, at the time of the Peninsular War, the castle was taken and held by the French for a time, and was manned by 50 French troops who used the Castle to control the main coastal road.

By the early 20th century, the Castle had fallen into disrepair and was occupied by homeless residents with up to 70 people living there until 1975 when it became a Guardia Civil barracks.

In 1994 the Manilva Town Hall acquired the building and it is currently occupied by the Department of Culture of the Town Council of Manilva, although there are currently plans to turn the Castle into a museum and visitor centre.