Born as the son of a merchant on 29 January 1773 in Gernrode/Harz. He studied mathematics, physics and chemistry at the University of Halle/Saale. He completed his studies in these disciplines as well as in mechanics at the Mining Academy at Freiberg/Sachsen. One of his teachers in Freiberg was the mineralogist and geognostician Abraham Gottlob Werner, who introduced Mohs to mineralogy and geognosy. In 1801, Mohs became a pit foreman at the Neudorf/Harz mine. He came to Vienna for the first time in 1802 in order to sort and annotate the important collection of the banker J.F. van der Nüll. (This collection was incorporated into the Imperial Mineralogical Collection in 1827). With his systematic classification of the realm of minerals, Mohs was in conflict with most of the other mineralogists of the time, due to his preference for physical attributes as the principles of classification (i.e., shape, cleavage, hardness, and specific weight), as opposed to the chemical composition of the minerals. In 1812, Mohs was appointed Professor of Mineralogy at the Joanneum in Graz. During this time, he developed the Mohs’s Scale of Hardness. His stay in Graz was only of short duration and in 1817, he succeeded his teacher, A.G. Werner, at the Mining Academy in Freiberg. In 1826, Mohs was appointed Professor of Mineralogy at the University of Vienna. However, he held his inaugural lecture at the Imperial Mineralogical Cabinet where in 1834, he also held the post of custodian concurrently with his professorship.

In 1835, Mohs left the Mineralogical Cabinet upon his appointment as mining counsellor at the Mining University in Leoben. Mohs died on 29 September 1839 during a trip to Italy.