“Constant improvement” is one of our mottos here at Betterview. Customers rely on our Property Intelligence platform to make crucial policy decisions, making a real impact on business decisions and on the lives of policyholders. Our engineering team takes that responsibility seriously, and they work hard to constantly improve our computer vision models. The more accurate our models, the more customers can gain a holistic understanding of real property risk, prevent future losses, and improve their expense ratio.
Transparency is another guiding principle at Betterview. For that reason, we are pleased to share a new report documenting the performance of our models. Read or download this document today to learn more about the accuracy of each of our classifiers, including roof condition attributes such as ponding, staining, rusting, and more. The report provides percentages on the precision and recall of each classifier, as well as an explanation of our team’s methodology. (This document will be updated as our Model Performance is documented over time; follow this link for the most up-to-date version.)
This report presents the statistical performance of Betterview’s proprietary computer vision models used in PropertyInsight. The numbers are calculated at the property level, meaning that they measure each model’s ability to detect the presence or absence of a condition or feature across an entire property.
The test samples used to compile these metrics were compiled using the following principles:
Balanced distribution - each dataset contains an even number of positive examples (images that contain the condition) and negative examples (images that do not contain the condition). This ensures that the performance metrics are comparable across models and not distorted by the relative frequencies of these different conditions in the real world. For example, structural damage is much less common “in the wild” than overhang; if we were to measure the performance on datasets that reflect this real-world distribution, the structural damage model would have much worse precision than overhang due solely to how rare it is.
Representative images - within each positive and negative set, examples are representative of the types of occurrences to occur in the real world. This is distinctly different than measuring performance on the dataset used to train the model, which typically contains a higher percentage of “challenging” examples (e.g., blue roofs for the tarp model) in order to effectively teach the model about these edge cases.
Sample size - each dataset contains a minimum of 1,000 images to ensure statistical significance of the metrics and minimize uncertainty.
We measure the performance of each model by comparing the model predictions to a ground truth dataset created by a team of expert labelers. The labelers have been trained using detailed technical documentation of each condition and thorough review of tens of thousands of images. Once trained, they identify these conditions through detailed visual inspection of the imagery.
We present the performance of each model using two complementary metrics: Precision and Recall:
Precision represents the percent of the time that the model prediction is correct when it says that a condition is present
Recall represents the percent of real instances of the condition that are picked up by the model
A higher precision means fewer false positives. A higher recall means fewer false negatives.
The uncertainty in each metric is also provided to convey the variability in results based on the sample used. A model with precision of 96% and uncertainty of ±2% can be interpreted as: we have 90% degree of confidence that the precision of the model is between 94% - 98%.
This table reflects the current model suite underlying Betterview’s PropertyInsight product. We routinely improve our models as we grow our training sets and advance our machine learning algorithms, and as we do we update this table to reflect the new versions.
* Performance metrics for these models are based on a dataset containing less than 1,000 images.
The Betterview platform provides P&C insurers with a holistic and actionable view of real property risk based on a combination of computer vision models and third-party property intelligence. Using our platform allows underwriters to take direct action to minimize risk at every stage of the policy lifecycle, from quoting to claims to renewal. 86% of underwriters report taking direct action after viewing the insights generated by our platform. Reach out today to discuss how Betterview can help your business!
The numbers in this post are based on the current Model performance at the time of posting. Follow this link to see the most up-to-date documentation of Betterview's Model Performance.