They spent the night at the palace and they have now gone to Catherine’s family in Bucklebury, Berkshire.
Other visitors to the palace included Prince Harry and Pippa Middleton.
After a morning of visitors, the duke and duchess were seen leaving Kensington Palace with their newborn at 13:00 BST.
The BBC’s royal correspondent Luisa Baldini said there had been wide speculation that the couple would head to the maternal grandparents’ home but it was not known how long they would stay.
Catherine is very close to her parents, Carole and Michael, and Prince William has previously said how comfortable he feels around them, she said.
“The house is large and private enough for the duke and duchess to bond with their newborn son,” she added.
The Queen visited without her husband, the Duke of Edinburgh, as he continues to recover from abdominal surgery.
She made the short journey from Buckingham Palace in a dark green Bentley.
The private meeting comes as the monarch prepares to leave for Balmoral in Scotland on Friday, where she has her traditional summer break.
At a reception at Buckingham Palace on Tuesday evening, the Queen said she was “thrilled” at the arrival of her third great-grandchild.
Louise Butt, a guest at the event for the Queen’s Award for Enterprise, said the monarch had also described the baby as a “big boy” and had said that “the first born is very special”.
There has been no word yet from the duke, who is expected to take about two months to convalesce following exploratory abdominal surgery last month.
The Queen – who also has two great-granddaughters – was not the first to meet the royal baby prince.
The Prince of Wales and Duchess of Cornwall, and Catherine’s parents, Carole and Michael Middleton, paid visits to the hospital on Tuesday afternoon.
Later that day, Catherine’s sister Pippa Middleton, accompanied by boyfriend Nico Jackson, went to Kensington Palace to meet her day-old nephew.
Kensington Palace said Prince Harry had also visited, but did not say when.
Prince Charles said the baby was “marvellous” while Mrs Middleton said he was “absolutely beautiful”.
Prince William was at the hospital for the birth at 16:24 BST on Monday, and stayed with his wife and son, who weighed 8lb 6oz (3.8kg), overnight.
They left at 19:15 BST on Tuesday – just under 27 hours after the baby’s arrival.
The couple described becoming parents as “very emotional”.
A Kensington Palace spokesman said on Wednesday it was “now private and quiet time for them to get to know their son”.
Although the couple have yet to decide on a name it is known that the newest royal will be known as HRH Prince (name) of Cambridge.
George is the bookmakers’ favourite for the first name of the prince, who is destined to be king one day. James and Alexander are among other names on a short price.
Asked by the BBC’s royal correspondent Peter Hunt if the baby had already been named George, Prince William replied: “Wait and see, Peter, wait and see.”
Royal fans waited seven days before the name of a newborn Prince William was announced in 1982, and there was a wait of a month following the Prince Charles’s birth in 1948.
All births in England, Wales and Northern Ireland, regardless of parentage, must be registered within 42 days at the hospital before leaving or a register office.