Bradley was appointed vice-chair for youth at Conservative HQ in January.
To date, he is best known for falsely accusing Labour leader Jeremy Corbyn of selling "British secrets to communist spies" in the 1980s. He later apologised for these accusations.
He wrote in his resignation letter: "If we do not deliver Brexit in spirit as well as in name, then we are handing Jeremy Corbyn the keys to Number 10."
Caulfield was appointed the Tory party's vice-chair fo women by May in January.
GONE: The news comes in the wake of a wave of resignations over the past few days
She openly campaigned for Britain to leave the EU, though her Lewes constituency voted to Remain by 52.1%.
In her resignation letter, she wrote: "This policy will be bad for our country and bad for the Party. The direct consequences of that will be Prime Minister Corbyn.
"Since the announcement on Friday my constituents, whether they voted leave or remain, have contacted me in large numbers to say they do not support the deal and it would therefore be more appropriate to have a vice-chair who can confidently defend the proposal."
Conservative chairman Brandon Lewis said: "Maria Caulfield and Ben Bradley have worked hard since the start of the year to promote women and young people on behalf of the Conservative Party.
"I wish them both well as they return to the backbenches to serve their constituents."
The news comes in the wake of a wave of Government departures, including former Brexit Secretary David Davis and former Foreign Secretary Boris Johnson.
Odds on next Prime Minister REVEALED
Who will be the next Prime Minister if Theresa May loses a confidence vote?
British Prime Minister Theresa May to leave post this year – 1/2
They were joined by another Brexit minister Steve Baker MP, and two parliamentary private secretary MPs Conor Burns and Chris Green.
Conservative leading body the 1922 Committee had a meeting to decide the Prime Ministers fate following the resignations.
But May appeared to have survived the day without a confidence vote being tabled.
At least 48 Tory MPs – 15% of the party's 316-strong representation in the Commons – must write to the chairman of the backbench 1922 Committee, Sir Graham Brady, to trigger a no-confidence vote.
Asked whether Mrs May would fight a no-confidence vote if one was called, a senior Number 10 source said simply: "Yes."
Labour leader Jeremy Corbyn said Mrs May should make way for a Labour administration if her Government cannot get its act together quickly.
Mrs May was heckled in Commons by Labour MPs, with shouts of “resign” and waves of “goodbye”.
The Chequers agreement "stands as a shattered truce, a sticking plaster over the Cabinet's cracks in this Government", Mr Corbyn told the Commons.
She was not left in the clear by her MPs either, with challenges from hardline Brexiteers – with Peter Bone claiming campaigners had been “betrayed”.
Brexit CHAOS: The Cabinet members ousted since the snap election
Prime Minister Theresa May has faced a series of Cabinet members leaving their positions since last June's snap election
Boris Johnson is the latest minister to quit since snap election
Mrs May has now appointed Dominic Raab and Jeremy Hunt to replace Mr Davis and Mr Johnson.
Tory digital minister Matt Hancock – known for his self-branded social networking app – will now takeover Hunts portfolio as Health Secretary.
The reshuffle is expected to continue as Mrs May attempts to stamp her authorities following the Chequers summit last Friday.