Goats are known far and wide for their ability to "test" fences. For us, we found our best option (short of a 10 ft brick wall) to be a well electrified roll of netting (PermaNet™ 10/48/6 shown).
Goats—Inquisitive CreaturesFolks who raise (or have raised) goats know that they are deceptively intelligent creatures. They have an enhanced ability to know when a fence is tired/worn or that the person milking them has absolutely no idea what they're doing. The goats use this knowledge to make themselves nuisances at the most inopportune moments.
|•||They breach fences when the roses in the garden are in full bloom or the in-laws are visiting.|
|•||They kick over the milk bucket for while novices try to milk them.|
To avoid such occurrences, read the articles below and brush up on how to milk a goat and use goats for browsing. Give the goats a job (and treat them well) and possibly they'll act more as eccentric pets rather than pests.