Zsuzsa Hegedus, who has known the nationalist Mr Orban for 20 years, described the speech as a “pure Nazi text”, according to Hungarian media.
The International Auschwitz Committee of Holocaust survivors called the speech “stupid and dangerous”.
Mr Orban’s spokesman said the media had misrepresented the comments.
The speech took place on Saturday in a region of Romania which has a large Hungarian community.
In it, Mr Orban said European peoples should be free to mix with one another, but that mixing with non-Europeans created a “mixed-race world”.
“We are willing to mix with one another, but we do not want to become peoples of mixed race,” he said.
Mr Orban’s anti-migration views are well known, but for Ms Hegedus Saturday’s speech crossed a line.
“I don’t know how you didn’t notice that the speech you delivered is a purely Nazi diatribe worthy of Joseph Goebbels,” she wrote in her resignation letter, according to the Hungarian hvg.hu news website.
Goebbels was the head of Adolf Hitler’s propaganda ministry.
Hungary’s largest Jewish group also condemned the speech and called for a meeting with Mr Orban.
Mr Orban’s remarks on race have been bitterly criticised by some in Hungary, and equally vehemently defended by others.
“Only one race inhabits this earth, Homo Sapiens. And it is unique and undivided,” chief rabbi Robert Fröhlich commented.
Opposition politicians, decisively defeated by Mr Orban’s Fidesz party in the April elections, said his remarks were “beyond the pale… unworthy of a European statesman”.
Government spokesman Zoltan Kovacs tried to dampen the growing chorus of condemnation, arguing that the prime minister had been outspoken on the topics of immigration and assimilation for years.
In the government flagship daily, Magyar Nemzet, an article praised Mr Orban for defending the idea of nationhood against a drive to mix all nations “into a grey, indistinguishable mass”.
At best, Mr Orban appears confused, sometimes speaking of the Hungarians as “the most mixed society”, at other times, appearing to suggest he believes in ethnic purity.
Zsuzsa Hegedus’s resignation is unlikely to have further repercussions in Hungary. Party discipline is tight, and resignations almost unheard of.
2px presentational grey line
Responding by letter to his longstanding adviser, Mr Orban defended his words.
“You know better than anyone that in Hungary my government follows a zero-tolerance policy on both anti-Semitism and racism,” he wrote.
His spokesman, Zoltan Kovacs, said the mainstream media was “hyperventilating about a couple of tough lines about immigration and assimilation”, but had stayed silent on the main points of the speech.
Mr Orban also spoke about the war in Ukraine, arguing that the West’s support of the country had failed, sanctions against Russia were not working, and that a negotiated peace deal should be the priority.
Despite receiving large amounts of EU funds, the Hungarian government led by Mr Orban frequently clashes with the EU over rule-of-law issues such as press freedom and migration.
Hungary’s prime minister has in the past been on good terms with Russia’s President Vladimir Putin, and is the only EU leader to openly criticise Ukraine’s President Volodymyr Zelensky.