TRIPOLI (Reuters) -- Libya's interior minister said Tuesday he could would not risk an armed confrontation with hardline Muslims behind a string of sectarian attacks, in an unusually candid admission of the scale of the security challenge facing the country.

Ultra-conservative Muslims bulldozed shrines sacred to Sufi Muslims in Tripoli and the western city of Zlitan on Friday and Saturday, stoking fears of the spread of factional violence following the fall of strongman Moammar Gadhafi.

The government is struggling to contain scores of armed militias, many of whom took part in the revolution that toppled Gaddafi and have refused to give up their weapons.

Interior minister Fawzi Abdel A'al initially offered his resignation on Saturday after he was criticized for not doing enough to stop the Tripoli assault, which took place in broad daylight in the center of the city as security forces looked on.

Abdel A'al on Tuesday said he had reconsidered and decided to stay in the job.

"When I submitted my resignation, I thought I would be relieving many people. But it seems my resignation will further complicate security and I withdrew it," Abdel A'al told reporters, without going into further details.

He defended the security forces decision not to confront the attackers and suggested the heavily-armed groups might out-gun government forces.

"If we deal with this using security we will be forced to use weapons, and these groups have huge amounts of weapons. We can't be blind to this. These groups are large in power and number in Libya. I can't enter a losing battle (with them), to kill people over a grave," he told reporters.

It was up to the country's religious bodies to stop the desecration, he said.

"If all shrines in Libya are destroyed so we can avoid the death of one person (in clashes with security), then that is a price we are ready to pay," the minister added.