Another 863 people were injured in the incident at Mina, which occurred as two million pilgrims were taking part in the Hajj’s last major rite.
They converge on Mina to throw stones at pillars representing the devil.
Preparations for the Hajj were marred when a crane collapsed at Mecca’s Grand Mosque this month, killing 109 people.
It is the deadliest incident to occur during the Hajj in 25 years.
Pilgrims travel to Mina, a large valley about 5km (3 miles) from Mecca, during the Hajj to throw seven stones at pillars called Jamarat, which represent the devil.
The pillars stand at three spots where Satan is believed to have tempted the Prophet Abraham.
People were going towards the direction of throwing the stones while others were coming from the opposite direction. Then it became chaotic and suddenly people started going down.
There were Nigerians, Nigeriens, Chadians and Senegalese among other nationals. People were just climbing on top of others in order to move to a safer place and that’s how some people died.
People were chanting Allah’s name while others were crying, including children and infants. People fell on the ground seeking help but there was no-one to give them a helping hand. Everybody seemed to be on their own.
It affected some members of our group. I lost my aunt as a result of the stampede and at the moment, two women from our entourage – a mother and her daughter – are still missing.
The Saudi civil defence directorate said in a statement that the stampede occurred at around 09:00 local time (06:00 GMT) at the junction of Street 204 and Street 223.
The pilgrims were walking towards the five-storey structure which surrounds the pillars, known as the Jamarat Bridge.
The incident happened when there was a “sudden increase” in the number of pilgrims heading towards the pillars, the statement said.
This “resulted in a stampede among the pilgrims and the collapse of a large number of them”, it added.
Security personnel and the Saudi Red Crescent were “immediately” deployed to prevent more people heading towards the area, the directorate said.
The Saudi health minister, Khaled al-Falih, said the crush occurred because pilgrims failed to follow directions.
He said “many pilgrims move without respecting the timetables” established by authorities.
2006: pilgrims die in a crush at foot of Jamarat Bridge in Mina
1997: pilgrims are killed when fire fuelled by high winds sweeps through Mina’s tent city
1994: pilgrims die in a stampede during the stoning ritual
1990: pilgrims, mainly Asian, die in a stampede in an overcrowded tunnel leading to holy sites
1987: people die when security forces break up an anti-US demonstration by Iranian pilgrims
The hundreds of wounded have been taken to four hospitals in the area by the more than 220 rescue vehicles sent to the scene.
Amateur video and photographs posted on social media showed the bodies of dozens of pilgrims on the ground. They were all dressed in the simple white garments worn during the Hajj.
The civil defence directorate said the victims were of “different nationalities”, without providing details.
Iran’s state news agency, Irna, said at least 43 Iranians were among the dead.
The UK Foreign Office said it was in contact with the local authorities and was urgently seeking more information about whether British nationals were involved.
Saudi-owned al-Arabiya TV reported that the head of the central Hajj committee, Prince Khaled al-Faisal, had blamed the stampede on “some pilgrims with African nationalities”.
But the head of Iran’s Hajj organisation, Said Ohadi, told Irna that two paths close to the scene of the incident had been inexplicably closed off by the Saudi authorities, resulting in the build-up in pilgrims.
What rituals do pilgrims perform? The pilgrimage takes place in several stages over five days, including circling the Kaaba (a cube-like building in the centre of the mosque) en masse and throwing seven stones at pillars called Jamarat which represent the devil.
How many people go? Well over a million pilgrims from outside Saudi Arabia, and several hundred thousand from inside the kingdom, converge on the site each year.
How do the authorities cope? Authorities deployed 100,000 security personnel and 25,000 extra health workers this year, as well as 100,000 air-conditioned tents for temporary accommodation.
The Saudi authorities have spent billions of dollars on improving transport and other infrastructure in the area in an attempt to try to prevent such incidents.
The Hajj is the fifth and final pillar of Islam. It is the journey that every able-bodied adult Muslim must undertake at least once in their lives if they can afford it.