Accuracy Evaluation Guide - Traffic Sensors

This guide helps customers evaluate the accuracy of Butlr’s Heatic technology for occupancy and activity monitoring in open areas using Traffic Mode sensors.

Butlr conducts rigorous internal testing before releasing any new feature. While internal tests simulate a variety of real-world scenarios, results may vary in actual deployments due to site-specific conditions such as layout, installation height, and ambient temperature.

We encourage customers to validate performance in their own environments. If you encounter any issues or discrepancies, Butlr’s support team is ready to assist with troubleshooting and site-specific optimization.

Before You Start Testing

All sensors installed onsite should have passed Deployment QA Guidelines. Before testing Butlr’s system, double-check the following key items:

Frame Rate
All Traffic Mode sensors should operate between 6.5–8 FPS. Notify your Butlr representative ahead of testing so they can verify the frame rate remotely.

Sensor Setup Matches Digital Twin

Confirm that the physical sensor setup aligns with your Butlr Studio Digital Twin, including:

• Correct sensor orientation
• Correct door line configuration
• Installation height below 9.2 ft (2.8 m)
• Wall-mounted or ceiling-mounted configuration

For more details, see General Sensor Installation: Aligning Physical Setup with Your Butlr Digital Twin in Studio.

System Behavior and Environmental Adaptation

Butlr’s thermal sensors adapt well to different environments and are more sensitive to temperature changes than to lighting or stillness. For accurate results, make sure the sample activity reflects real-world usage.

System Strengths

• Works equally well in light or dark
• Unaffected by furniture changes, as long as Studio matches physical layout

Operational Guidelines for Best Accuracy

• People should be at least 60 cm (2 ft) apart
• Optimal ambient temperature: 18–29°C (65–85°F)
• Expected data latency: ~10 seconds

Edge Cases & Known Limitations

While Butlr’s algorithm auto-adjusts for most anomalies, the following may occasionally affect accuracy:

False Positives

• Sudden or strong sunlight shifts
• Large animals in the detection zone
• People continuously standing on virtual door line

False Negatives

• People wearing thick clothing or helmets
• Body temperature matching ambient temperature (rare)
• Individuals spaced less than 60 cm (2 ft) apart
• Large objects blocking the sensor’s view
• Poor connectivity in device mesh or weak internet signal

Best Practices for Testing Traffic Sensor Accuracy

Evaluating traffic sensor accuracy is more complex than presence sensing, especially when collecting ground truth data in live environments. There are two primary methods for evaluating performance:

• In/Out Traffic Accuracy
• Occupancy Count Accuracy

Recommended Approach: In/Out Traffic Accuracy

In most use cases, traffic sensors are deployed at floor or zone entrances. We recommend validating performance at the sensor level using in/out traffic counts, which provide a clear and repeatable testing method.

• Choose a single entrance monitored by one sensor.
• Collect ground truth using camera footage (preferred) or manual counts to minimize human error.

Why use this method:

• Clear accuracy targets and repeatable standards
• Easier to isolate issues and track improvements
• Recommended for both internal QA and customer validation

Occupancy Count Accuracy (Derived from Traffic)

Using traffic sensors to estimate space-level occupancy is less consistent and harder to validate. Accuracy can vary significantly depending on environmental and statistical factors.

Limitations to consider:

• Sensitive to edge cases and low sample sizes
• Accuracy drifts over time without reset or correction
• Relies on consistent baseline occupancy
• Performance varies based on the number and placement of sensors

This method is typically used for floor-level insights but should be interpreted with caution.

For scenarios where space-level occupancy is critical (e.g., large multi-purpose rooms or bathrooms), refer to [this guide] for best practices on collecting occupancy ground truth and comparing it to traffic-derived estimates.

Getting Reported Traffic Counts from the Butlr Dashboard

To evaluate a Traffic Sensor, first locate the corresponding floor or room in the Dashboard. Then follow these steps:

1. Set the toggle to "Entrance" in the Floor/Room Overview.
2. Click "Show Breakdowns" to display in/out traffic counts.
3. Adjust the time window to match your test period using the dropdown or by zooming in on the chart.
4. Hide other Traffic Sensors in the same space to isolate the data from the sensor you're testing.
5. Click the download icon to export the CSV file.
6. Use the sensor’s MAC address to identify it in the exported data.

Sample Testing and Dataset

You can evaluate sensor accuracy through either performed testing or real-world data collection, depending on your time and vendor evaluation protocols. If you don’t have a defined testing procedure, refer to the guidance below. A sample dataset—with accuracy calculated using traffic counts within each test window—is also available for reference.

Understanding Traffic Sensor Counting Logic

By default, Butlr’s Traffic Sensors treat any trace with an equal number of entries and exits (e.g., 1 in and 1 out within the same trace) as 0 in / 0 out. This logic helps prevent overcounting in cases such as U-turns or loitering, which are common in high-traffic areas.

Note on Simultaneous Movements:
When people enter and exit at the same time, these are usually recorded as separate traces and are counted normally. However, in rare cases, if one person enters and another exits almost immediately through the same or nearby pixels, their paths may be mistakenly connected into a single trace.

Testing Guidance:
When performing controlled tests, ensure that each subject fully exits the sensor’s coverage area before starting a new test event. This ensures the system generates distinct traces and the reported traffic counts match your expectations.

Keep in mind that in real-world environments, some movement patterns may be grouped into a single trace and excluded from traffic counts to maintain accuracy. Occupancy data, however, will still reflect presence correctly.

Performed Testing

If time allows, aim for at least 50 entries and 50 exits to ensure a statistically meaningful sample.
Include a mix of scenarios based on the entrance type—e.g., single-person passes, multi-person passes, and varying walking patterns—to reflect real-world usage.

Real-World (Random) Testing

For busier entrances where real-time counting is difficult, we recommend recording video during the test period and determining ground truth by reviewing the footage afterward.