🎭 Part 2 — Count Me Out & Assert Me Wrong: Two Sneaky Playwright Pitfalls

Hey friends! 👋 Remember how we talked about relying on our e2e tests and synthetic tests as quality gates for daily releases in part #1? Well, here’s the hard truth: Stability doesn’t come for free. 💸

We had to put in a ton of work to make sure our test suites weren’t crying wolf every other day. Because let’s be real — nobody wants to be dragged into a 🔥 grilled 🔥 call just because a flaky test blocked half the company’s release pipeline! (And yes, the false alerts cost? Painful. 💔)

So, mission clear: Fail only when there’s a real issue — not because of some sneaky test instability.

Misunderstood Playwright Behaviors!🤯

Here’s the kicker — some of our flakiness came from misreading how Playwright’s functions actually work. And honestly? It’s so easy to get tripped up, especially when function names are… a little misleading. 😅

🔍 Scenario #1: The .count() Deception

Problem: We needed to get the number of items on a page. Simple, right? First instinct?

let totalCount = page.locator('[data-testid="listItems"]').count();

False sense of security: “Hmm, maybe it’s async?” So we awaitit:

let totalCount = await page.locator('[data-testid="listItems"]').count();

But… surprise! 🎭 .count() does NOT wait for all elements to load.

• It returns immediately with whatever’s in the DOM at that moment.
• If your list is still loading? Boom. Flaky count.

Our reaction: 😱 “Wait, WHAT?!”
After digging through docs (and GitHub issues), we realized:

.count() is not your patient friend. It’s that one colleague who replies ‘Done!’ before actually checking.

🚨 The .all() Trap (Plot Twist!)

“Okay, fine! What about .all()? That should wait for everything, right?"

NOPE. 🙅‍♂️

• Same issue! .all() also doesn’t wait for all elements to exist.
• It just grabs what’s available right now and gives you the result.

💡 The Fix? Wait for Stability!

Instead of blindly trusting .count() or .all(), we forced stability by:

• Waiting for a specific count (e.g., await expect(await page.locator("[data-testId='listItems']")).toHaveCount(5)).
• Or, even better — waiting for at least some items (e.g.,await expect(await page.locator("[data-testId='listItems']").count()).toBeGreaterThanOrEqual(5)).

“Assertions Are Straightforward, Right?” (Spoiler: NOPE.)

You’d think assertions are the least confusing part of testing. WRONG. Playwright has two types of assertions, and if you mix them up? Hello, flakiness my old friend. 😭

Here’s the kicker:

1. Auto-Retry Assertions — The patient heroes. They keep trying until success (or timeout). And they return promises!
2. Non-Retry Assertions — The impatient ones. They pass/fail immediately. No promises, no retries, no mercy.

💥 Scenario #2: The Deadly Assertion Mix-Up

We thought we fixed our .count() problem with:

expect(await locator.count()).toBeGreaterThan(5);

BUT SURPRISE! 🎭

• .count() doesn’t wait (we know this now).
• .toBeGreaterThan() is a non-retry assertion—so it also doesn’t wait!

Result? A flaky mess. Again. 🤦‍♂️

🚨 The Real Problem: Sync + Async = Chaos

We needed:

• Get the count (but wait for stability).
• Assert the count (but keep retrying until valid).
• Timeout gracefully (no infinite loops, please).

Translation: We needed to convert a non-retry assertion into an auto-retrying one!

💡 The Fix? expect.toPass() to the Rescue!

Playwright gives us two magic tools for this:

• expect.toPass() – Wraps any assertion in a retry loop.
• expect.poll() – Lets you poll custom logic with retries.

Here’s what saved us:

await expect(async () => {
  expect(await page.locator('[data-testid="listItems"]').count())
    .toBeGreaterThan(5);
}).toPass({ timeout: 2 * 60 * 1000 });

What’s happening here?
✅ Retries the entire block until it passes (or hits timeout).
✅ No more flakiness from partial loads!
✅ Explicit timeout control (because defaults won’t save you).

🔥 Pro Tip: Set Your Timeout Wisely!

⚠️ toPass()default timeout? ZERO. 😱
Always set it to n × expected load time (e.g., 2*60*1000for 2 mins).

🎯 Key Takeaways:

• .count() ,.all() ,.isVisible() and.isHidden()are impatient. They don’t wait.
• Always pair them with explicit waits to avoid flakiness.
• False alerts = wasted time + trust erosion. Fight flakiness like it’s your job! (Because… it kinda is.)
• Not all assertions retry! Know which ones do
• Combine non-retry assertions with toPass() for bulletproof waits.

Flaky tests stealing your sanity? Drop your horror stories below! 👇 😅

Tags: Playwright, Testing, E2E Testing, Test Automation, Flaky Tests, Software Quality, Web Development