Rescue operations are under way and police reinforcements have been sent.
Senior state police officer SK Bhardwaj said it was difficult to say how many people had died as many bodies had been dismembered.
The incident took place at 08:40 India time [03:10GMT]. The pilgrims were hit by the Rajya Rani Express travelling on the opposite track.
Railway officials said the express train was travelling at high speed as it was not expected to stop at Dhamara Ghat station.
But after the accident, it stopped a few hundred metres away. An angry mob then pulled out the driver and severely assaulted him.
More police have been sent to the area but the region is extremely remote and inaccessible by road.
Bihar Chief Minister Nitish Kumar has expressed grief over the incident and ordered district officials to the scene.
India’s state-owned railway network is vast – it operates 9,000 passenger trains and carries some 18 million passengers every day.
A government-appointed safety panel in its report last year said about 15,000 people were killed each year crossing train tracks in what officials describe as “unlawful trespassing”.