Arsenal would have provisionally supplanted leaders Chelsea on goals scored with victory and took a 20th-minute lead when Alexis Sanchez's free-kick deflected in off Williams.
But Seamus Coleman equalised with a 44th-minute header before Williams's first Everton goal ended his team's five-game winless run and sent Ronald Koeman's men up to seventh in the table.
"We showed two faces," Everton manager Koeman, who had defender Phil Jagielka sent off in stoppage time, told BT Sport.
"This was a good match though because we showed how we have to play -- with commitment and aggression. You see the reaction of the crowd when you go for it and that's what we did."
Arsenal manager Wenger said the game had been "very physical".
"We faced many physical challenges and that disturbed our game, but we were a bit unlucky to lose as well," he said.
"They were a bit scared at first, but they got their confidence. We did fight, I cannot fault that."
Arsenal made only one change from their 3-1 win over Stoke City, fit-again right-back Hector Bellerin replacing injured centre-back Shkodran Mustafi, and took a 20th-minute lead.
The goal stemmed from a comedy of errors on the edge of Everton's box as Ross Barkley lost possession, Williams fouled his own team-mate Idrissa Gueye and Jagielka tripped Francis Coquelin.
Sanchez sent the free-kick curling towards Everton goalkeeper Maarten Stekelenburg and in seeking to clear the ball, Williams succeeded only in deflecting it into his own net.
Everton did not look like equalising, with Romelu Lukaku and Aaron Lennon slicing shots off target, but with half-time looming they drew level.
Leighton Baines curled a right-foot cross towards the back post from the left and Coleman nipped in between Laurent Koscielny and Nacho Monreal to glance a deft header past Petr Cech.
The noise level rose as the game wore on and with four
minutes to play Everton struck the winner, Williams flying in at the back post to send a thudding header past Cech from Barkley's corner.
It was Arsenal's first league defeat since their 4-3 loss to Liverpool on the season's opening day and means they will be six points off the pace if Chelsea win at bottom club Sunderland on Wednesday.
Leicester remain without back-to-back league wins this season after defeat at Bournemouth -- their eighth of the campaign -- kept them in 14th place, just four points above the relegation zone.
"The result is too severe against us because we created more chances than Bournemouth," said Leicester manager Claudio Ranieri. "If we continue to fight at this level, sooner or later we will get a good result."
Eddie Howe's Bournemouth leap four places to eighth, nine points clear of the bottom three.
You’ve reached your limit of {{free_limit}} free articles this month.
Subscribe now for unlimited access.
Already subscribed? Log in
Subscribe to read the full story →
Smart Quarterly
₹900
3 Months
₹300/Month
Smart Essential
₹2,700
1 Year
₹225/Month
Super Saver
₹3,900
2 Years
₹162/Month
Renews automatically, cancel anytime
Here’s what’s included in our digital subscription plans
Exclusive premium stories online
Over 30 premium stories daily, handpicked by our editors


Complimentary Access to The New York Times
News, Games, Cooking, Audio, Wirecutter & The Athletic
Business Standard Epaper
Digital replica of our daily newspaper — with options to read, save, and share


Curated Newsletters
Insights on markets, finance, politics, tech, and more delivered to your inbox
Market Analysis & Investment Insights
In-depth market analysis & insights with access to The Smart Investor


Archives
Repository of articles and publications dating back to 1997
Ad-free Reading
Uninterrupted reading experience with no advertisements


Seamless Access Across All Devices
Access Business Standard across devices — mobile, tablet, or PC, via web or app
