Months of rising rhetoric about religious freedom came to a head on Tuesday when a city commission voted that a contentious plan to build a mosque two blocks from “Ground Zero” could move forward.

The proposal to demolish an old Burlington Coat Factory building and replace it with a 13-storey Islamic centre has reopened the wounds of September 11 2001 and provoked some of the most heated anti-Muslim rhetoric since the attacks on the World Trade Center.

On Tuesday, the New York City Landmarks Preservation Commission voted unanimously that the proposed site of the centre, which includes a mosque, should not be given protected landmark status, a label that would have limited the scope of its development.

“It’s hurtful to the memories of those people who died there,” said Andy Sullivan, a construction worker from Queens who was across the street from the towers during the attacks and lost friends that day. After the meeting, Mr Sullivan vowed that New York’s construction workers would never build the mosque.

Developers and opponents of the mosque alike had anxiously awaited the ruling from the landmarks commission to determine the fate of the project, originally called Cordoba House and recently rebranded Park 51 after weeks of criticism. The $100m project hinged on whether the existing structure was considered historically significant, meaning it could be refurbished but not demolished.

About 100 people gathered in an auditorium near Ground Zero to hear the verdict, with a mix of cheers and tears erupting when the commissioners said the mosque could move forward. The stone-faced building with Italian palazzo columns was built in 1858 but was not deemed exceptional enough to warrant city protection.

In spite of the backlash, publicity surrounding the project has made it more likely to succeed. Soho Properties, which owns the building, and Park 51, the non-profit organisation that will be its tenant, say new backers have been “knocking down our door” to make donations.

“The controversy has had a negative impact on us as individuals,” said Daisy Khan, spokeswoman for Park 51 and the wife of Faisal Abdul Rauf, the mosque’s future imam. “We’re still moving ahead with the planning and now it will be easier for us to do a national fundraising campaign.”

Rick Lazio, a New York Republican running for governor, has accused the imam of sympathising with terrorists and called for an investigation into the funding for its construction.

A few seats away from Ms Khan at the meeting, Linda Rivera held a sign that read “Islam builds mosques at the site of their conquests”. Another nearby protester screamed “Trojan horse”.

National politicians have also latched on to the debate, with Sarah Palin calling the mosque a “stab in the heart” of the families of victims, Newt Gingrich declaring it an “insult” and Rudy Giuliani, New York’s former mayor, billing it a “desecration”. A July Quinnipiac University poll shows 52 per cent of New York voters opposed.

Backers of the centre say it is intended to be the Muslim equivalent of a Jewish community centre or YMCA, offering fitness facilities, exhibitions and lectures to promote better understanding of other cultures. According to Sharif El-Gamal, chief executive of Soho Properties, the location was picked to serve the growing Muslim population in lower Manhattan.

“Those who oppose this project need to understand more about Islam,” said Dalia Mahmoud of the Muslim Public Affairs Council. “The actions of a few terrorists don’t speak for a whole religion.”