The Brunnenburg, also called Prunberg, was built by the family Tarant who had their main residence at the castle Burg Tarantsberg (or Dornsberg) in the valley Vinschgau. The Brunnenburg used to consist of a quadratic keep at the highest point of the castle and a simple palas at the unassailable side near the mountain.

The Tarant called themselves by the castle shortly after and as shown by Wilhelm Tarant de Prunnenperch, mentioned in 1253. By marriage, the castle was owned by Utlein von Schlandersberg from 1268 onwards. During the siege of the Schloss Tirol by Karl von Böhmen in 1347, the castle was probably heavily damaged, although this is only an assumption. The following decades had many different owners for the Brunnenburg and in 1457 it fell to the family Kripp. Since when the castle was no longer inhabited and therefore completely derelict is not clear but around 1600 it was already talked about as being old and ruinous.

In the early 20th century, the building was bought by Karl Schwickert who then went on to the restore it costly. Around 1905, the Brunnenburg was rebuilt in its historicising style, though the fortifications of the past were no longer evident. The construction did not stand for long as the western porch fell into the Köstengraben in 1922.

The castle Brunnenburg became famous because of the poet Ezra Pound, who lived here from 1958 onwards.