The Scottish Conservative member served as third Presiding Officer of the Scottish Parliament from 2007 until 2011.
Sir Alex was first elected to the Scottish Parliament in 1999, standing down before the last Holyrood elections in 2016.
He was knighted for services to politics and public life in 2016.
Party leader Ruth Davidson described him as a "proper gentleman" who will be mourned by all who knew him.
SIR ALEX FERGUSSON: Scotland is in mourning after the death of the politician
RIP: Scottish Conservatives paid tribute to Sir Alex Fergusson
Ms Davidson said: "Alex Fergusson was a proper gentleman. His decency, compassion and wisdom earned him friends across all parties. His passing will be mourned by all who knew him.
"He wasn't just a major figure in the Scottish Conservatives, he was also, as the Scottish Parliament's third Presiding Officer, a hugely significant figure at Holyrood and a stout defender of the Parliament. He enjoyed the respect of every MSP in the chamber."
She added: "On behalf of everyone in the Scottish Conservatives, I offer my deepest condolences to his family and friends."
Scottish Secretary David Mundell said: "To me, Alex was foremost a great friend and I will miss him dearly.
"He is the most thoroughly decent person I have ever known. He was the absolute opposite of the career politician and was guided a sense of public duty and his core values of family, community and country, rather than party dogma."
Before being elected to Scottish Parliament, Sir Alex ran a hill farm in South Ayrshire.