The virus has killed almost 121,000 people across Africa and infected 4.18 million.
J&J unit Janssen Pharmaceutica NV has entered into a deal with the African Vaccine Acquisition Trust (AVAT) to deliver 220 million doses of its single-dose shot.
AVAT could order an additional 180 million doses through 2022.
The deal follows months of negotiations with the AU, which announced a provisional agreement in January to buy 270 million doses of vaccines from J&J, AstraZeneca and Pfizer-BioNTech.
The status of the talks with the other two companies is not known.
J&J's vaccine came to the market much later than those of AstraZeneca and Pfizer but has recently gained widespread acceptance globally, especially in Africa.
"J&J requires just a single dose, it makes it a very good programmatically to rollout," said John Nkengasong, director of the Africa Centres for Disease Control and Prevention.
He said the price of the J&J's dose is likely to be $10.
In February, South Africa put the use of AstraZeneca's shot on hold after data showed it gave minimal protection against mild-to-moderate infection caused by the country's dominant variant.
Several countries in Europe have suspended using AstraZeneca's COVID-19 vaccine as they investigate a small number of reports of rare blood clotting in people who got the shot. Global regulators have said the shot is safe and effective.
Pfizer's shot has more complex storage and transportation needs than other vaccines, making it more challenging to deploy in warmer climates or in poorer countries.
AVAT said on Monday that many of the AU's 55 member states had shown a strong preference for J&J.
Africa is also grappling with a more-infectious variant identified in South Africa amid concerns about delays in deliveries of AstraZeneca shots via the COVID-19 Vaccines Global Access Program (COVAX) that is aimed at supplying poorer countries. The continent's first COVAX shipment of COVID-19 vaccines was received in Ghana less than five weeks ago on Feb. 24.
The continent is far behind nations, including Israel, the United States and Britain in its vaccination rollout. World Health Organization (WHO) Director-General Tedros Adhanom Ghebreyesus called the divide "grotesque" when talking about differing access to shots between rich and poor nations. Almost half of Britons have received their first dose, while in contrast, only 0.4% of South Africa's population has received one dose.
"We need to immunize at least 60% of our population in order to get rid of the virus from our continent. The J&J agreement enables us to move towards achieving this target," Nkengasong said.
Most of the supplies will be produced by Aspen Pharma in South Africa, AVAT said in a statement.
President Cyril Ramaphosa said on Monday that South Africa will get 30 million vaccines from Aspen's facility while a total of roughly 250 million will be distributed across the continent from the facility.
Aspen has contracted with J&J to manufacture 300 million doses.
As part of the AU vaccine plan, the African Export-Import Bank (Afreximbank) has approved up to $2 billion in finance for countries to buy shots via the AU.
Europe approved J&J's single-dose vaccine earlier this month. The United States, Canada and Bahrain have also approved the shot.
Late last year, J&J said it and Gavi the vaccine alliance expected to enter into a deal that would provide up to 500 million doses of the vaccine to COVAX through 2022.