The US President replaced him with the director of the CIA, Mike Pompeo.
US President Donald Trump on Tuesday announced the departure of his top diplomat Rex Tillerson, to be succeeded by the current CIA chief Mike Pompeo.
A senior US official said the president wanted to reshuffle his team ahead of talks with North Korea.
"Mike Pompeo, Director of the CIA, will become our new Secretary of State. He will do a fantastic job!" Trump tweeted.
"Thank you to Rex Tillerson for his service!" he added.
The US president announced the appointment of Gina Haspel, a career intelligence officer, to head the Central Intelligence Agency — the first woman tapped for the post.
Trump praised Pompeo, a former US army officer and congressman who led the CIA for nearly 14 months, as "the right person for the job at this critical juncture."
"He will continue our program of restoring America's standing in the world, strengthening our alliances, confronting our adversaries, and seeking the denuclearization of the Korean Peninsula," Trump added.
The reshuffle comes days after the spectacular announcement of a meeting between Trump and North Korean leader Kim Jong Un, whose date and details have yet to be determined.
Trump called for Pompeo to be swiftly confirmed in the role — as his administration heads into the high-stakes talks.
He had scant words of praise for Tillerson, who was traveling in Africa when Trump made the snap decision last week to accept an invitation to meet Kim.
After the announcement, Tillerson suspended his schedule on grounds he was "unwell" and eventually cut short his trip to fly back to Washington.
"I want to thank Rex Tillerson for his service," Trump said. "A great deal has been accomplished over the last fourteen months, and I wish him and his family well."
The former Exxon CEO has had a rocky tenure as the top US diplomat, and had long been rumored to be about to be pushed out.
Tillerson was repeatedly forced to deny he had fallen out with Trump — vowing to remain in post despite a sensational report that he once dubbed the president a "moron."
A respected figure in the oil business, the 65-year-old Texan's tenure at the State Department drew scorn from Trump's opponents, from former diplomats and from the Washington policy elite.
During his time in post, he was faced with an extraordinary array of foreign policy challenges, from North Korean nuclear threats to Russian subversion to attacks on US diplomats in Cuba.
But his efforts were often overshadowed by Trump's un-diplomatic style and his streams of taunting tweets stirring international tensions.