Seeing the Light of Day in Construction: Perception's Role in Delivering Autonomy
Aerial photo of the operator-less construction site at Dimensions 2022. Photo Credit: Matt Sewick
The Autonomy Journey
Autonomy continues to see limited availability in construction relative to its market size. Delivering autonomous solutions to the construction market requires a combination of capabilities including perception, path planning, machine control and more to produce a robust and reliable system capable of meeting operational requirements.
Within autonomy, the role of perception presents many unique challenges, especially in unconstrained, off-road environments. Perception systems must accommodate a wide range of operational design domains (ODDs) to provide the situational awareness required for a safe and reliable autonomous machine. Because of the dynamic nature of a construction site, many machines will have unique perception requirements. For example, an excavator must be able to continuously produce terrain maps as the machine works and perceive how material is moving over time, while a dozer must be able to perceive material build-up on its blade.
Perception: Enabling Situational Awareness for Autonomous Machinery
Perception’s main differentiating factor from other system components in the autonomy stack is that it interprets the natural environment into a structured version of the environment a machine can understand. This is done so that downstream components, such as control and path planning, can then function using this structured version of the current state of the environment. Construction applications’ required operation in unstructured environments means that no two scenes are the same, regardless of whether or not the ODD or application is constrained. The very essence of construction is to effect change on the environment only adding to the variation in the operating environment. Effective machine perception requires algorithms to interpret sensor measurements in real-time. The machine must be told where an object is, how the object moves over time, where the object is expected to go, what the object is, what objects are the same across time frames, the structure of the surrounding environment, traversability of the ground terrain and more. How these tasks are accomplished depends on the sensor modalities and architecture employed.
Trimble Autonomy enabled autonomous compactor shown at Dimensions 2022. Photo Credit: Matt Sewick
Safety, Safety, Safety
Safety is the top priority for any modern construction site. With advancements in autonomy, the goal is to one day be at a point where workers can coordinate and monitor machinery remotely. With reliable, predictable and communicating autonomous machinery, we believe we can have an impact on enhancing the safety of a live construction site. I saw this firsthand when a manually operated water truck performing dust suppression entered an area being compacted by Trimble’s autonomous compactor. The operator of the water truck was initially concerned when he saw there was no operator present in the cab of the compactor. However, after observing the predictable movement of the autonomous compactor across the site, he began coordinating his movement with it, clearly highlighting the potential for collaboration with autonomous machinery.
Construction machines are often large machines with inherent and substantial blind spots, posing risks on the construction site. Unlike an operator with eyes focused on a limited region, perception sensors and associated perception software have the ability to continuously monitor the entirety of the surrounding environment, even in cases where obscurants block the visual field of view (FOV). Furthermore, autonomous construction machines can operate in hazardous environments and conditions that would put people at risk. We have seen this in cases such as compaction near the edge of an open pit or berm where rollover safety is a concern. Without a human operator inside the cab of the compactor, compaction can take place closer to hazardous terrain with reduced costly mitigations as the concern of human injury is negated.
Operator blindspots shown for various construction machinery. Figure Credit: CAT, CDC
Beyond the Line of Sight
The classic perception paradigm involves processing raw sensor measurements, producing detections from those measurements and then associating and tracking detections to produce the best estimate of object motion. We know that perception for autonomous machines allows them to see beyond the line of sight of an operator by continuously monitoring the FOV around machinery, using obscurant penetrating sensors to see through dust and employing accurate GNSS positioning to know exactly where the machine is.
Perception goes beyond what can be seen to what can be felt and heard. An excavator operator may receive tactile feedback through the controls or the seat when the bucket hits a hard object while an experienced dozer operator may feel vibrations or hear changes in engine sound when the blade is not set optimally. Leveraging perception data even allows autonomous machines to take proactive safety measures by matching historical accident data with the current perceived situational information. This results in a shift from reactive measures to events transpiring, to implementing proactive measures that can prevent situations that have historically been shown to result in accidents. As we move towards increasingly autonomous construction sites, we must expand the scope of perception beyond what can be seen to what can be felt, heard and understood.
Autonomous excavator digs a trench at Dimensions 2022. Photo Credit: Matt Sewick
Impact on Efficiency and Productivity
In addition to safety, the impact of perception for autonomous construction machinery is also about productivity and efficiency. Today 68% of US contractors are asking workers to do more as a result of skilled labor shortages and 56% of contractors report a challenge in meeting project schedules due to skilled labor shortages. Autonomous construction machinery, equipped with advanced perception capabilities, can operate around the clock with minimal downtime. At Trimble’s Dimensions user conference in Las Vegas in November 2022, severe winds and blowing sand caused our offsite location to be temporarily closed to visitors. However, the unfazed autonomous compactor equipped with inherently weather-robust radar sensors and ruggedized hardware continued to operate without concern. Visitors had the opportunity to view the compactor in operation from the comfort of the conference hall. One could easily see this translating to site managers remotely organizing and deploying autonomous machinery from the comfort of their office. Additionally, accurate positioning provided by Trimble’s GNSS technology, the same technology powering hands-free driving, means that work is done right the first time regardless of factors such as weather conditions or operator fatigue. This reduction of rework directly translates to cost savings.
The Site Ahead
While there have been significant advancements and even early adoption of autonomy in construction, the journey towards a fully autonomous construction site is still ongoing, largely due to the advanced perception capabilities required to perform complex construction tasks. The role of perception in autonomous construction is not only foundational for performing construction tasks safely but also transformative, poised to change the way construction work is done. We are confident these increases in perception capabilities will result in improved safety, efficiency and productivity that are greatly welcome to an industry with some of the tightest margins and schedules.