Israeli police said the three gunmen reached one of the gates near the al-Aqsa compound, opened fire and fled towards al-Aqsa Mosque where they were shot dead by police officers on Friday.
Israeli police later ordered the closure of the compound, saying there would be no prayers at the site on Friday.
Al Jazeera's Harry Fawcett, reporting from Jerusalem, said the Palestinians reportedly shot at Israeli security forces near the Lion's Gate to the Old City.
"The three attackers were armed with two machine guns, a pistol and a knife, according to Israeli police," Fawcett reported.
"They were then pursued inside the al-Aqsa Mosque compound. In the courtyard of the compound, a final gun battle ensued between the gunmen and Israeli security forces.
"Management of al-Aqsa mosque said the bodies of two Palestinians were inside the courtyard of the compound," Fawcett reported.
Palestinian President Mahmoud Abbas spoke to Israeli Prime Minister Benjamin Netanyahu by phone on Friday and condemned the shooting, Palestinian news agency Wafa said.
"The president expressed his strong rejection and condemnation of the incident that took place at al-Aqsa mosque, as well as his rejection of any violent incidents from any side, especially in places of worship," Wafa said.
Abbas called on Netanyahu to end the shutdown of the holy site. Netanyahu assured Abbas the long-standing status quo at the compound, which gives Muslims exclusive prayer rights, would be maintained, Wafa said.
'Tensions could rise'
The ancient, marble-and-stone compound houses the al-Aqsa Mosque, Islam's third-holiest site, and the 7th-century Dome of the Rock. Thousands pray there every Friday.
The western wall of the compound, also known as the Wailing Wall, is considered the holiest site in Judaism.
"It is the first time in years that the compound closes for prayers on Friday. Of course, this has the potential to increase tensions among the about 10,000 Palestinians who normally come here for worship," Fawcett said.
Following the incident, access to the Old City was restricted.
Dozens of Palestinian residents waited in the sun to be allowed into the Old City.
Um Nidal, who lives in Bab Huta - a neighbourhood in the Muslim quarter - told Al Jazeera that she has been waiting for more than three hours with her child, who is 16 months old.
"I've been waiting here since 11:30am, the police will not let me in and I was pushed away," she said.
Adham Asfour, 30, also said he was unable to return home.
"We have no choice but to wait here until they let us through," he told Al Jazeera.
With the mosque closed, hundreds of Palestinians instead held prayers near Damascus Gate.
Police arrested Muhammed Ahmad Hussein, the grand mufti of Jerusalem and city's highest Islamic authority, after he led prayers, said Azzam el Khatib, director of the religious organisation Al Awqaf.
Hussein had earlier decried the mosque closure.
"Forbidding the Friday prayer is an unfair procedure," Sheikh Omar Keswani, a religious official at al-Aqsa, told Al Jazeera.
"What happened earlier is now being taken advantage of by the Israeli right to impose a new reality in al-Aqsa Mosque," Keswani added.
Haj Khalil Abu Elsheikh, 77, travelled 100km from Beersheba to attend Friday prayers at Al-Aqsa Mosque.
"No religion allows this," he told Al Jazeera, referring to praying on the side of the road. "No belief accepts this."
In a separate incident on Friday, Israeli forces shot dead an 18-year-old Palestinian during a raid in Dheisheh refugee camp south of Bethlehem in the occupied West Bank.
Israeli forces arrested two Palestinians during the raid, sources told Al Jazeera.
Since September 2015, Israeli forces have killed more than 254 Palestinians, Israel claims most of them were attackers.
In that same period, Palestinian attackers have killed 43 Israelis, two visiting Americans and a British tourist.
Author: Ibrahim Husseini