Alexis Tsipras, whose bloc came second, aims to put together a cabinet that will reject austerity measures imposed as part of the EU’s bailout deal.
But analysts say his attempts, after Sunday’s poll, are also unlikely to achieve the necessary numbers.
French and Italian voters also rejected austerity candidates in Sunday polls.
France’s socialist President-elect, Francois Hollande, said in his victory speech that he would seek an alternative to austerity.
And results from local elections in Italy indicate a marked swing away from mainstream parties. An anti-euro protest movement led by comedian Beppe Grillo made significant inroads in Parma and Genoa.
In Greece, both the centre-right New Democracy and former coalition partners Pasok, the traditional parties of power, saw their support drain away in favour of radical parties on the left and right.
Reacting to the election results, German Chancellor Angela Merkel said austerity measures were “not negotiable” and described Greece’s reforms as of “utmost importance”.
Markets slumped following the election results in France and Greece, but largely recovered later. The Athens stock exchange, however, had plunged 6.67% by the end of Monday.
In return for two EU/IMF bailouts worth a total of 240bn euros (£190bn; $310), Greece agreed to make deep cuts to pensions and pay, raise taxes and slash thousands of public sector jobs.
Mr Tsipras went into talks on Tuesday with President Karolos Papoulias, who was expected to give him three days to try to form a coalition.
His first meeting afterwards is expected to be with the leader of Democratic Left, although he is likely to talk to all the party leaders, except the ultra-nationalist Golden Dawn.
New Democracy leader Antonis Samaris, whose party topped the poll, tried and failed to put together a coalition on Monday. His party beat Syriza by a slim margin, but under Greek rules, has a 50-seat bonus in parliament for coming first.
The 38-year-old Syriza leader has already promised to stitch together a left-wing cabinet to reject the “barbaric” measures associated with the bailout deal.
“We will exhaust all possibilities to reach an understanding, primarily with the forces of the left,” Mr Tsipras said.
But the numbers just do not seem to add up, the BBC’s Matthew Price in Athens says.
Mr Samaras stressed on Monday night that his party had done “everything possible” to form a government.
“I tried to find a solution for a government of national salvation, with two aims: for the country to remain in the euro and to change the policy of the bailout by renegotiation,” he said in a televised address.
“We directed our proposal to all the parties that could have participated in such an effort, but they either directly rejected their participation, or they set as a condition the participation of others who did not accept.”
If Mr Tsipras fails to clinch a coalition deal, the mandate then passes to Pasok leader and ex-finance minister Evangelos Venizelos.
Mr Venizelos said Syriza and the smaller leftist party of Fotis Kouvelis should be involved in any new coalition.
“It is necessary for the government of national unity to include all the forces that have a pro-European outlook,” Mr Venizelos said. “The minimum level of agreement is that Greece remains in the euro.”
Snap elections will be called if no agreement is found, with analysts saying the polls could be held as early as next month.
Under Greece’s current bailout plan, billions of euros in further austerity cuts will have to be found in June – and the country is also counting on a 30bn euro (£24bn; $39bn) instalment in EU/IMF funds.
Despite emerging as the biggest party, New Democracy’s support slipped from 33.5% in the last election to less than 19% on Sunday.
Support for the centre-left Pasok, which also supported the austerity measures, plummeted from 43% to just over 13%.
Syriza took 16.8%, while fellow anti-bailout party Golden Dawn, won almost 7%.
The financial chaos has sparked huge social unrest, and led to a deep mistrust of the parties considered to be the architects of austerity.