IS began demolishing the site, which was founded in the 13th Century BC, on Thursday, according to Iraqi officials.
The head of the UN’s cultural agency condemned the “systematic” destruction in Iraq as a “war crime”.
IS, which controls large areas of Iraq and Syria, says shrines and statues are “false idols” that have to be smashed.
“They are erasing our history,” said Iraqi archaeologist Lamia al-Gailani.
Nimrud lies on the Tigris river, about 30km (18 miles) south-east of Mosul, which IS controls.
Many of the artefacts found there have been moved to museums in Baghdad and overseas, but many remain on site.
BBC Middle East correspondent Jim Muir says the attempt to destroy Nimrud is already being compared with the Taliban’s demolition of theBamiyan Buddha rock sculptures in Afghanistan in 2001.
As well as destroying artefacts, Islamic State also trades in them – and the trade is one of its key sources of revenue.
IS “assaulted the historic city of Nimrud and bulldozed it with heavy vehicles,” the tourism and antiquities ministry said on Thursday.
It said the militants continued to “defy the will of the world and the feelings of humanity”, calling for a UN Security Council meeting to discuss how to protect cultural heritage in Iraq.
Nimrud covers a large area, and it is not yet clear whether it has been totally destroyed, our correspondent says.
But a local tribal source told Reuters news agency: “Islamic State members came to the Nimrud archaeological city and looted the valuables in it and then they proceeded to level the site to the ground.
“There used to be statues and walls as well as a castle that Islamic State has destroyed completely.”
Ancient Assyrian city on the River Tigris Capital of Assyria for about 150 years.
First excavations in modern times undertaken by Europeans starting in the 1840s Treasures unearthed included sections of royal palaces, individual statues and smaller artefacts Investigations stopped for decades but in 1949 Sir Max Mallowan (husband of writer Agatha Christie) began fresh excavations.
Extensive photographic record of remaining treasures made in the 1970s Irina Bokova, head of the UN cultural agency Unesco, condemned the assault.
“This is yet another attack against the Iraqi people, reminding us that nothing is safe from the cultural cleansing under way in the country,” she said.
“The deliberate destruction of cultural heritage constitutes a war crime. There is absolutely no political or religious justification for the destruction of humanity’s cultural heritage.”
Dr Gailani told the BBC: “Nimrud for us in Iraq and for me as an archaeologist is one of the most important [sites]. There are still quite a lot of things that are standing – the reliefs and the statues, the famous winged bulls.
“They are erasing our history. I wish it was a nightmare and I could wake up.”
It is a “horrific crime against the cultural heritage of the whole world”, said the president of the British Institute for the Study of Iraq, John Curtis, calling Nimrud “the best preserved Assyrian site in Iraq”.
Last week, IS released a video apparently showing militants with sledgehammers destroying historic artefacts in a museum in Mosul.
One militant was seen drilling through and pulling apart what appeared to be a stone winged bull.
In the video, the artefacts are described as “false idols” and their destruction justified in religious terms.
That attack was also condemned by the UN as a war crime.
IS has controlled Mosul, Iraq’s second largest city, and nearby areas since June 2014 – a region with nearly 1,800 of the country’s 12,000 registered archaeological sites.
Iraqi forces and Shia militia, have united to try to drive IS militants from the northern city of Tikrit, with the support of Iran.
The militants are also being targeted by a US-led coalition mounting air strikes.
The latest strikes came on Thursday, with the coalition targeting IS positions around Mosul and other Iraqi cities, as well as several locations across the border in Syria.