It is as the town of Stillwater that we must consider much of the early history of the Columbus, the change in name not taking place until January of 1894. As early as 1875 Horace Countryman, C.H. Countryman and W.H. Norton settled at a point two miles west of the present town and opened a trading post. In 1877 when the mail and stage line was established through the Yellowstone Valley, Horace Countryman was the proprietor of the stage station. A post office named Stillwater was also established and he then became the postmaster.

When the Northern Pacific railroad came through the valley in 1882 a station was established at the present site of the town of Columbus, and this too was named Stillwater. At that time the post office was moved to the new town of Stillwater. Horace and C.H. Countryman established a ferry to cross the Yellowstone and were the proprietors of a hotel. Norton had engaged in a general store business as did J.I. Allen. Horace Countryman eventually platted the town site in 1889.

The Northern Pacific railroad officials and the 150 people of the town of Stillwater were inconvenienced by the delivery of mail to Stillwater, Minnesota because of the similarity of the abbreviations of the names, so a change of name was decided. NP Superintendent Dorsey suggested the name of Columbus. A petition was sent to the post office department and the name officially changed to Columbus on January 1, 1894. After many failed attempts for the development of a new county named Roosevelt, the formation of Stillwater County was completed in 1913 from portions of Carbon, Sweet Grass and Yellowstone County and Columbus became the county seat. The campaign to develop Stillwater County encountered some bitter opposition from the county seats of Red Lodge and Big Timber, however; Yellowstone County’s seat, Billings, failed to protest and sponsored the new county as an asset.