Zambada, 76, founded the criminal organisation with Joaquin "El Chapo" Guzman, who is currently jailed in the US.
Arrested with Zambada on Thursday was Guzman's son, Joaquin Guzman Lopez, said the US justice department.
In February, Zambada was charged by US prosecutors with a conspiracy to make and distribute fentanyl, a drug more powerful than heroin that has been blamed for the US opioid crisis.
In a written statement on Thursday evening, US Attorney General Merrick Garland said the two men lead "one of the most violent and powerful drug trafficking organizations in the world".
"El Mayo and Guzman Lopez join a growing list of Sinaloa cartel leaders and associates who the Justice Department is holding accountable in the United States," Mr Garland said.
"Fentanyl is the deadliest drug threat our country has ever faced, and the Justice Department will not rest until every single cartel leader, member, and associate responsible for poisoning our communities is held accountable,” added Mr Garland, the top law enforcement officer in the US.
American prosecutors say the Sinaloa cartel is the biggest supplier of drugs to the US.
US authorities have previously noted that fentanyl is the leading cause of death for Americans aged 18 to 45.
The US Drug Enforcement Administration (DEA) had been offering a reward of up to $15m (£12m) for Zambada's capture.
During Joaquin "El Chapo" Guzman’s trial in 2019, his lawyers accused Zambada of bribing the “entire” Mexican government in exchange for living openly without fear of prosecution.
"In truth he controlled nothing," Guzman’s lawyer, Jeffrey Lichtman, told jurors of his client. "Mayo Zambada did," he claimed.
According to the US state department, Zambada is also the owner of several legitimate businesses in Mexico, including “a large milk company, a bus line, and a hotel”, as well as real estate assets.
Alongside fentanyl charges, he is also facing charges in the US ranging from drug trafficking, murder, kidnapping, money laundering and organised crime.
In May, Zambada’s nephew - Eliseo Imperial Castro, who was known as "Cheyo Antrax" - was killed in an ambush in Mexico. He was also wanted by US authorities.
Zambada is arguably the biggest drug lord in the world and certainly the most influential in the Americas.
He had evaded authorities for decades, and as such, his arrest has come as a shock in Mexico.
Details of the arrests of the two men remain unclear, but it appears they flew into the United States.
Citing Mexican and US officials, the Wall Street Journal reports that Zambada was tricked into boarding the plane by a high-ranking Sinaloa member following a months-long operation by Homeland Security Investigations and the FBI.
Zambada believed he was going to inspect clandestine airfields in Mexico, the paper reports.
In a statement, US Secretary of Homeland Security Alejandro Mayorkas said the Sinaloa cartel "pioneered the manufacture of fentanyl and has for years trafficked it into our country, killing hundreds of thousands of Americans and devastating countless communities".
FBI director Chris Wray said the arrests are "an example of the FBI's and our partners' commitment to dismantling violent transnational criminal organizations like the Sinaloa Cartel," he said.
As more information emerges, Zambada's arrest will no doubt be heralded by President Joe Biden's administration as one of the most significant operations by the DEA in years.
Zambada co-founded the Sinaloa cartel in the wake of the collapse of the Guadalajara cartel at the end of the 1980s.
While Joaquin "El Chapo" Guzman was the public face of the organisation and the most notorious of the two men, many believed it was in fact El Mayo who was its real leader.
Not only ruthless, he was also innovative, creating and maintaining some of the earliest links with Colombian cartels to flood the US with cocaine and heroin.
And more latterly, fentanyl.
His leadership of the criminal empire has endured in the face of changing presidents in Mexico and the US, amid repeated anti-drug offensives from successive governments and constant efforts by his enemies in other drug-trafficking organisations to bring him down.
That is no mean feat in the violent, dangerous and treacherous underworld in which he has operated as an unassailable kingpin for many years.
Yet that extraordinary resilience appears to have run out in El Paso, Texas – a city blighted by the influx of the synthetic opioid, fentanyl, much of which was smuggled in by his organisation.
Max Matza & Will Grant, Mexico correspondent