The ex-foreign secretary also described burkas as “absolutely ridiculous” and said their wearers looked like “bank robbers”.
Johnson came out against calls for a ban on the face-covering veils in public places – but compared their wearers to rebellious teenagers.
Conservative Party chairman Brandon Lewis has asked Johnson to apologise for comments he made in his column in the Telegraph.
In a message on Twitter, Mr Lewis said: "I have asked Boris Johnson to apologise."
But a source close to Johnson branded the demands “ridiculous” and insisted by not discussing burkas “we are simply yielding ground to extremists”.
The insider said: "It is ridiculous that these views are being attacked – we must not fall into the trap of shutting down the debate on difficult issues.
"We have to call it out. If we fail to speak up for liberal values then we are simply yielding ground to reactionaries and extremists."
Daily Star Online contacted the Tories to ask what will happen in Johnson fails to say sorry.
BORIS JOHNSON: The top Tory sparked outrage with his burka comments
BURKA: Boris Johnson said Muslim women in veils look like 'letter boxes'
“I have asked Boris Johnson to apologise”
Johnson wrote: "If you tell me that the burka is oppressive, then I am with you.
"If you say that it is weird and bullying to expect women to cover their faces, then I totally agree – and I would add that I can find no scriptural authority for the practice in the Koran.
"I would go further and say that it is absolutely ridiculous that people should choose to go around looking like letter boxes."
He added he felt "fully entitled" to ask women to remove face coverings, and called for schools to be able to do the same if a student "turns up … looking like a bank robber”.
TORY: Boris Johnson claims women wearing burkas look like 'bank robbers'
Johnson's comments were widely criticised by the Muslim community and MPs.
The call followed a hail of criticism for Mr Johnson's remarks, which were described as "offensive" by Foreign Office minister Alistair Burt and "bigotry" by former Tory chair Baroness Warsi.
Labour MP Jess Phillips said she would report Mr Johnson to the Equality and Human Rights Commission.
Meanwhile, Labour MP David Lammy branded him a "pound-shop Donald Trump".
The assistant secretary general of the Muslim Council of Britain, Miqdaad Versi, said Mr Johnson was "pandering to the far right".
APOLOGISE: Conservative Party chairman Brandon Lewis called for Boris Johnson to say sorry
While agreeing with Mr Johnson that the burka should not be banned, Mr Burt told BBC Radio 4's Today programme: "I would never have made such a comment, I think there is a degree of offence in that, absolutely right."
And Lady Warsi accused Mr Johnson of adopting the "dog whistle" tactics of former Donald Trump aide Steve Bannon in the hope of attracting support from right-wing Tories for an eventual leadership bid.
Mr Lammy added: "Muslim women are having their burkas pulled off by thugs in our streets and Boris Johnson's response is to mock them for 'looking like letter boxes'.
"Our pound-shop Donald Trump is fanning the flames of Islamophobia to propel his grubby electoral ambitions."
GAFFE-PRONE: Boris Johnson is known for his often outrageous remarks and controversial comments
Denmark last week followed France, Germany, Austria and Belgium in banning face-covering garments such as the burka and niqab in public places.
A fine of around £120 has already been imposed on a woman wearing a niqab in a shopping centre in the town of Horsholm, after another woman reportedly tried to tear it off.
Repeating her call for an independent inquiry into Islamophobia in the Conservative Party, Lady Warsi told Channel 4 News: "Muslim women should not be a useful political battleground for Old Etonians.
"It is crass and it must stop, and it must be condemned by the leadership right from the Prime Minister down."