Oliver's "Billy and the Epic Escape", released in May, contains a passage where an Indigenous Australian girl living in foster care is abducted by the story's villain - a sensitive issue in a country where Indigenous children were for decades forcibly removed from their parents.
It also contained errors made by mixing different Indigenous languages.
The Guardian newspaper reported on Sunday Oliver apologised for the offence caused by the book.
"I am devastated to have caused offence and apologise wholeheartedly," Oliver, who is currently in Australia promoting his latest cookbook, said in a statement.
"It was never my intention to misinterpret this deeply painful issue. Together with my publishers we have decided to withdraw the book from sale."
Oliver's publisher Penguin Random House did not immediately respond to a request for comment.
Sue-Anne Hunter, who sits on a government commission into injustices against Indigenous people in the state of Victoria, called the depictions in the book "insensitive".
"The publication of Jamie Oliver's children's book represents a deeply concerning example of how Indigenous people continue to face misrepresentation and cultural appropriation in mainstream media," she said in a social media post.
Among the world's oldest cultures and together speaking hundreds of distinct languages, Australia's Indigenous communities have suffered centuries of discrimination since the country was colonised by Britain in the late 18th century.
Thousands of Indigenous children were forcibly taken from their parents and placed in care or with white families as late as the 1970s, in a policy now known as the "Stolen Generation".
