Originally known as An Rudha, "the point" or "the Ness", Invergordon received its name from Sir William Gordon, a local landowner of the early eighteenth century. For centuries the estate on which Invergordon now stands was known as Inverbreakie (the mouth of the Breakie); the Breakie presumably being the stream which enters the Firth at Rosskeen Bridge near the old parish church. The earliest mention of Inverbreakie occurs in the thirteenth century when the Castle, about a mile inland from the Ness, was occupied by a Fleming; placed there it is said by William the Lion. From a very early period there were a few thatched houses near the spot where the harbour now is; these were known as "The Ferry Houses".
The Castle and the estate were purchased by Sir William Gordon around the beginning of the eighteenth century. During his lifetime the Castle, originally a modest dwelling, was considerably enlarged and at the same time the policies were improved and plans laid for the building of a town near the Ness. William was succeeded by his son, Sir John, who may be regarded as the real founder of the town and his ambitious plans included industrial development, but he was hampered by inherited debt. Sir John died, childless, in 1773. This said Sir John had a bust carved by Esme Bouchardon, one of the 18th Century's most fashionable sculptors; the bust was carved while Bouchardon studied in Rome and undertook private commissions for many of the great of Europe visiting the city.
In 1780 the Invergordon estate was purchased for Robert III Macleod of Cadboll. With the coming of the Macleod's of Cadboll to the district the development of Invergordon went on apace. A harbour was built and very soon the village, for it was still only a village, became the principal distributing port in the north. When the old Castle was destroyed by fire, a new and very handsome mansion was built by the late Robert Bruce Aeneas Macleod in 1872. Some years earlier the Castle policies had been transformed into a thing of beauty, the "American Gardens" as they were called, with their profuse display of rhododendrons, stately trees and flower-bordered walks being famed throughout the country. The developing community led to the formation of a Police Burgh in 1864; Robert Bruce Aeneas Macleod of Cadboll became its first Provost. When the estate was sold about the time of the First World War, the glory of the Castle and the Gardens disappeared, and in 1928 the Castle itself was demolished.