Chile’s government criticised Mr Ledezma for wearing the clothing, which bore a logo referring to Bolivia’s claim to a stretch of coastline it lost to Chile in a 19th-Century war.
Mr Ledezma was delivering drinking water to a flood-hit region of Chile.
He was replaced in the post by former university rector Reymi Ferreira.
Me Ledezma travelled to the northern Chilean town of Copiapo on Tuesday aboard a Bolivian Air Force plane to deliver drinking water.
Copiapo and the surrounding region has been devastated by flash floods triggered by the heaviest rains to fall in 80 years.
At least 23 people died and 57 are missing and many communities were left without electricity and drinking water.
During his visit, Mr Ledezma wore a top sporting the motto “The sea belongs to Bolivia” and a map in the colours of the Bolivian flag including the disputed area.
Bolivia has been landlocked for 136 years, ever since it lost 400km (250 miles) of coastline to Chile during the War of the Pacific.
Bolivia has long tried to reclaim the territory and has repeatedly severed relations with Chile when those attempts failed.
Two years ago, it filed a lawsuit against Chile at the International Court of Justice in The Hague. The court has not yet made a ruling.
While handing out water to flood victims, Mr Ledezma told local radio he was there not to provoke but to help.
He said he put on the vest after he got cold.
He said he often wore it and shared the sentiment of the motto printed on it, but that it was not a topic he had intended to raise on his visit to Chile.
Chile’s Foreign Minister Heraldo Munoz said that while his country was grateful for the help “this tragedy should not be used for political purposes”.
Bolivian President Morales apologised to Chile saying it had been Mr Ledezma’s personal decision to wear the vest.
“We can’t commit these type of errors. No minister is autonomous and there’s no room for personal decisions,” he said at the swearing-in of the new minister.