BRUSSELS— Facebook Inc. has hit a new obstacle in Europe in its planned takeover of a small New York-based startup, showing how antitrust muscle-flexing is increasingly affecting deals an ocean away.

With a new interpretation of an old law, European Union competition regulators have given themselves sweeping authority to review merger cases that previously would have escaped their notice, sparking outcry from companies and their lawyers that the change will cause confusion in deal-making.

The European Commission, the EU’s competition watchdog, said late Monday it was using its new authority to review Facebook’s proposed takeover of Kustomer, a startup specializing in customer-service platforms and chatbots, announced in November.

Margrethe Vestager, the commission’s antitrust chief, who announced the new approach in March, said it is necessary to let her team assess and potentially block dominant companies from buying up smaller players to eliminate would-be rivals. Such deals are known as killer acquisitions because they can snuff out competition.

Some antitrust lawyers say the new policy, which required no regulatory or public review, introduces uncertainty and lacks legal rigor. Even those less overtly critical say it remains unclear how the commission will use its new powers.

“We’re going into a period of discovery, which is quite unnerving,” said Alec Burnside, a partner with law firm Dechert LLP in Brussels.

Takeover deals have long faced scrutiny in jurisdictions far from their home markets. Chinese review of U.S. chip maker Qualcomm Inc.’s proposed takeover of Dutch peer NXP Semiconductors NV in 2018 helped doom that deal. The EU in 2001 opposed General Electric Co. ’s proposed acquisition of Honeywell Inc., killing their plans.

In the digital age, when companies can reach markets everywhere, competition concerns have increased in far-flung markets that once might have had minimal say on a deal.

“On the question of what kind of deals cause problems, views have shifted,” said Mr. Burnside.

For U.S. competition authorities, who cooperate closely with EU counterparts, the bloc’s new approach has already proven useful. The Federal Trade Commission in March sued to block life-sciences company Illumina Inc.’s planned $7.1 billion acquisition of medical-detection test maker Grail Inc., announced in September.

Facebook CEO Mark Zuckerberg.

Facebook CEO Mark Zuckerberg.

Photo: Mark Lennihan/Associated Press

Ms. Vestager’s office in April announced a preliminary review of the Illumina-Grail deal using its new powers, and last month announced an in-depth investigation of the takeover, potentially bolstering the FTC’s case.

Illumina and Grail are fighting the FTC in a trial that starts later this month and are challenging the EU review.

Facebook’s takeover of Kustomer didn’t cross the commission’s traditional threshold for review based on revenue. Terms weren’t disclosed, but people familiar with the deal said it valued Kustomer at slightly more than $1 billion.

The deal did trigger a review in Austria, which in spring requested an EU review along with other member countries. The commission on Monday announced an in-depth investigation of the deal, which can take 90 days, saying it was concerned that the deal could reduce competition in the customer-relationship management software market.

“We will continue to fully cooperate with the European Commission’s review,” said a Facebook spokesperson. “The transaction is pro-competitive and will bring more innovation to businesses and consumers.”

The commission’s ability to review both deals stems from its new interpretation of one article of its merger-review law, adopted in 1989. At the time, not all countries in what was then the European Communities had their own merger laws. Article 22 allowed members to refer cases to Brussels for scrutiny.

As the EC, and later the EU, expanded its antitrust experience and powers over time, it asked countries not to send it cases, to avoid being inundated.

In March Ms. Vestager reversed that guidance and asked members to refer small deals they found concerning, particularly in tech, pharmaceuticals and financial services.

Ms. Vestager said in an interview the change was necessary because the longstanding revenue-based merger review standard increasingly failed to catch significant acquisitions. Money has been replaced as the critical factor in many deals by a target company’s data, technology or its potential role after a merger.

After a four-year review of merger policies and past cases, she said, her team realized “actually, we do have what we need. We just need to increase our cooperation with the national competition authorities.” EU national merger-review standards can differ from those used in Brussels, targeting lower revenue levels and other standards.

“The fact that we have the necessary legal basis already, of course, makes things much easier,” Ms. Vestager said.

Lawyers experienced in EU competition law say the new approach and the commission’s explanation of it leave many unanswered questions, including about jurisdiction between member states and Brussels, over timing of merger reviews and whether all deals must be notified to EU authorities.

“They found this clever trick to expand their jurisdiction without changing a comma or consulting on it,” said Jay Modrall, a partner at law firm Norton Rose Fulbright in Brussels. “It creates a lot of legal and practical issues they’re sweeping under the rug.”

Commission officials have rejected such accusations, arguing any legal uncertainty always existed. They have pledged to use the new approach selectively, making it preferable to a proposed alternative of lowering the revenue threshold for deal review, which would have indiscriminately snared far more companies.

Ms. Vestager said the majority of merger deals will undergo simplified review. “We don’t want to hold anything up that would not pose a competition problem,” she said.

Write to Daniel Michaels at daniel.michaels@wsj.com