Accuracy Evaluation Guide - Traffic Sensors

This guide helps customers evaluate the accuracy of Butlr’s Heatic technology for occupancy and activity monitoring in open areas using Traffic Mode sensors.

Butlr conducts rigorous internal testing before releasing any new feature. While internal tests simulate a variety of real-world scenarios, results may vary in actual deployments due to site-specific conditions such as layout, installation height, and ambient temperature.

We encourage customers to validate performance in their own environments. If you encounter any issues or discrepancies, Butlr’s support team is ready to assist with troubleshooting and site-specific optimization.

Before You Start Testing

All sensors should have passed the Installation Sign-Off QA - Traffic Mode, completed by your Butlr installer or certified installation partner before leaving the site. Before testing Butlr’s system, double-check the following key items:

Frame Rate
All Traffic Mode sensors should operate between 6.5–8 FPS. Notify your Butlr representative ahead of testing so they can verify the frame rate remotely.

Sensor Setup Matches Digital Twin

Confirm that the physical sensor setup aligns with your Butlr Studio Digital Twin, including:

• Correct sensor orientation
• Correct doorline configuration — this is critical for accurate traffic counts. See the Doorline Adjustment Guide for Traffic Sensors before testing.

• Installation height within the recommended range for your mount type:

  • Wall mount (105° tilt angle): 6.9–9.5 ft (2.1–2.9 m)
  • Ceiling mount: 7.2–9.8 ft (2.2–3.0 m)

  • Installing sensors outside the recommended range may lead to less accurate results. Use Studio to preview sensor coverage at different installation heights.

• Wall-mounted or ceiling-mounted configuration

For more details, see General Sensor Installation: Aligning Physical Setup with Your Butlr Digital Twin in Studio.

System Behavior and Environmental Adaptation

Butlr’s thermal sensors adapt well to different environments and are more sensitive to temperature changes than to lighting or stillness. For accurate results, make sure the sample activity reflects real-world usage.

System Strengths

• Works equally well in light or dark
• Unaffected by furniture changes, as long as Studio matches physical layout

Operational Guidelines for Best Accuracy

• People should be at least 60 cm (2 ft) apart
• Optimal ambient temperature: 18–29°C (65–85°F)
• Expected data latency: ~10 seconds

Edge Cases & Known Limitations

While Butlr’s algorithm auto-adjusts for most anomalies, the following may occasionally affect accuracy:

False Positives

• Sudden or strong sunlight shifts
• Large animals in the detection zone
• People continuously standing on virtual door line

False Negatives

• People wearing thick clothing or helmets
• Body temperature matching ambient temperature (rare)
• Individuals spaced less than 60 cm (2 ft) apart
• Large objects blocking the sensor’s view
• Poor connectivity in device mesh or weak internet signal

Best Practices for Testing Traffic Sensor Accuracy

Evaluating traffic sensor accuracy is more complex than presence sensing, especially when collecting ground truth data in live environments. There are two primary methods for evaluating performance:

• In/Out Traffic Accuracy
• Occupancy Count Accuracy

Recommended Approach: In/Out Traffic Accuracy

In most use cases, traffic sensors are deployed at floor or zone entrances. We recommend validating performance at the sensor level using in/out traffic counts, which provide a clear and repeatable testing method.

• Choose a single entrance monitored by one sensor.
• Collect ground truth using camera footage (preferred) or manual counts to minimize human error.

Why use this method:

• Clear accuracy targets and repeatable standards
• Easier to isolate issues and track improvements
• Recommended for both internal QA and customer validation

Occupancy Count Accuracy (Derived from Traffic)

Using traffic sensors to estimate space-level occupancy is less consistent and harder to validate. Accuracy can vary significantly depending on environmental and statistical factors.

Limitations to consider:

• Sensitive to edge cases and low sample sizes
• Accuracy drifts over time without reset or correction
• Relies on consistent baseline occupancy
• Performance varies based on the number and placement of sensors

This method is typically used for floor-level insights but should be interpreted with caution.

For scenarios where space-level occupancy is critical (e.g., large multi-purpose rooms or bathrooms), refer to [this guide] for best practices on collecting occupancy ground truth and comparing it to traffic-derived estimates.

Getting Reported Traffic Counts

There are two ways to retrieve reported traffic counts, depending on whether you're testing in near-real time or reviewing a past test period.

Option 1: Live Testing with Studio (Near-Real Time)

Use Studio to view minute-level In/Out counts while testing is in progress:

1. Open Studio and navigate to the floor where your test sensor is installed.
2. Click the Traffic Sensor you're testing. The Sensor Settings panel will open.
3. Scroll to the Traffic Count section to see near-real-time In/Out counts at 1-minute intervals.
4. Match these counts against your ground truth for each test window.

Note: There may be a slight delay before counts appear, so allow a brief buffer before comparing against ground truth.

Option 2: Historical Dashboard (Yesterday or Earlier)

Historical data becomes available starting the day after collection. To review counts for a past test period:

1. In the Dashboard, go to the Historical tab and set a custom time range covering your test period.
2. Select a small time interval (e.g., 1 min) and switch to the By Sensor view.
3. Identify your test sensor by its MAC address in the chart legend.
4. Retrieve the counts in any of these ways:
   • Hover over the bars to see In/Out counts for each interval
   • Click the download icon to export a CSV
   • Use Query to pull the data programmatically

Tip: Match the time interval to your test event length. For example, if each test event lasted 5 minutes, use the 5-min interval so that count distribution across minute boundaries doesn't create discrepancies against your ground truth.

Sample Testing and Dataset

You can evaluate sensor accuracy through either performed testing or real-world data collection, depending on your time and vendor evaluation protocols. If you don’t have a defined testing procedure, refer to the guidance below. A sample dataset—with accuracy calculated using traffic counts within each test window—is also available for reference.

Understanding Traffic Sensor Counting Logic

By default, Butlr’s Traffic Sensors treat any trace with an equal number of entries and exits (e.g., 1 in and 1 out within the same trace) as 0 in / 0 out. This logic helps prevent overcounting in cases such as U-turns or loitering, which are common in high-traffic areas.

Note on Simultaneous Movements:
When people enter and exit at the same time, these are usually recorded as separate traces and are counted normally. However, in rare cases, if one person enters and another exits almost immediately through the same or nearby pixels, their paths may be mistakenly connected into a single trace.

Testing Guidance:
When performing controlled tests, ensure that each subject fully exits the sensor’s coverage area before starting a new test event. This ensures the system generates distinct traces and the reported traffic counts match your expectations.

Keep in mind that in real-world environments, some movement patterns may be grouped into a single trace and excluded from traffic counts to maintain accuracy. Occupancy data, however, will still reflect presence correctly.

Performed Testing

If time allows, aim for at least 50 entries and 50 exits to ensure a statistically meaningful sample.
Include a mix of scenarios based on the entrance type—e.g., single-person passes, multi-person passes, and varying walking patterns—to reflect real-world usage.

Real-World (Random) Testing

For busier entrances where real-time counting is difficult, we recommend recording video during the test period and determining ground truth by reviewing the footage afterward.