Safety & Security
Camera based ADAS for mass deployments
This is to satisfy new deployments as high volume standard equipment vs. the moderate customer option business as seen today. Analog Devices recently announced a new ADAS Processor Family, the Blackfin ADSP-BF60x series which was developed for this emerging mass market requirement. The first two processors of this new family are available now to service this new requirement for camera based solutions.
Depending on the ADAS tasks to solve, different types of sensors are being deployed in systems today. For the environmental near and far-field monitoring around the vehicle, Radar-, Ultrasonic-, LIDAR-, PMD-, Camera- and Night camera sensors are used either as single or multi sensor systems. The latter combine the sensor data to achieve more accurate results as desired. Camera systems will further see an additional boost, looking inside the car for analyzing the driver's state. Driver situation and prediction analysis will be used to better filter the various warning messages an ADAS system can generate in the future. The goal is not to overflow the driver with irrelevant warnings. If the system detects that a sporty driver is concentrating on the traffic in front of him, a premature warning of a car he is following might be very annoying. However, knowing that the driver is distracted while playing with his mobile phone or simply is about to fall asleep, an early enough warning or even braking action in the same situation might be very appropriate.
ADAS systems providing the extended view outside of the car while detecting and classifying objects, plus systems determining the driver state are all aiming at increasing road safety and will soon become part of the standard equipment in new cars. Increased driving comfort and economical aspects like controlling the timely gear shifts before an incline will additionally drive customer acceptance and will enable car manufacturers to market optional functionality in addition to the standard ADAS equipment.
ADAS deployments fueled by Euro NCAP
Euro NCAP tests and publically reports the safety of vehicles marketed in Europe. Transparent test methods are used while four categories are considered: Adult Occupant protection, Child Occupant protection, Pedestrian protection and Safety Assist (e.g. seatbelt reminder). As a result, Euro NCAP awards up to five safety stars which then will be published on their web site. Each safety level (one to five stars) requires a minimum number of points in each of the four categories and, in addition, a minimum number of total points.
It is expected that by 2017 cars will not be able to achieve the desired five star rating without ADAS present and that car manufacturers will therefore include at least one ADAS system as standard equipment by that time. While looking at the Euro NCAP rating scheme and comparing the changes over the years, one can get a feel for the current priorities. Between 2011 and 2012, requirements to achieve the minimum points for pedestrian protection were upped by 50%. This boots the importance for ADAS cameras due to their ability to not only detect and classify objects as pedestrians but for their capability to do the same with partially obscured pedestrians as required by the rating schemes.
Cameras make the standard equipment
Camera based ADAS is not new, the technology had some time to mature over the past years, but was mostly seen as a customer option on premium brand cars. This accumulated knowhow is now very useful while going after the emerging Euro NCAP requirements. However, a rethinking has to happen as the standard equipment business brings the commercial aspect into the focus. While in the past complex and high performance systems were developed, the task at hand is now a different one. ADAS systems must now support the “just required” functionality at the appropriate commercial level. System providers and component suppliers like Analog Devices are now asked to walk a fine line between enabling commercially attractive solutions while maintaining the freedom for OEM's to differentiate.
Enabled by the early engagement in driver assistance systems and the continued investment into this emerging technology, Analog Devices' has developed an advanced, dedicated ASAD solution and is now sampling the first two processors focusing on camera based systems. From the start, the specification for these ADAS Camera centric processors targeted the total cost of ownership (ToC) without taking the flexibility out of the hands of the system suppliers and OEM's. Next to the required programmability and enough processing power, lowest power consumption in class was achieved in order to keep the thermal design manageable. Support for functional safety following ISO26262 and the availability of an application oriented development environment with optimized vision processing libraries helps to design a total system with “time to market” and low risk in mind.
Optimum Performance on System Level
The ADSP-BF60x family reduces the overall cost of a five function system of up to 30%. The ADSP-BF609 (processing up to Mega Pixel format) and the ADSP-BF608 (processing up to VGA format) support up to five concurrent Vision functions with up to 30 processed frames per second. With the <1.3W at ambient temperature of 105°C, the ADSP-BF60x offer the lowest power consumption figures in their class.
To enable this, Analog Devices deployed a straight forward but unique concept. It is based on two Blackfin cores as they are used in production ADAS systems today. However, algorithms that can’t be modeled by software in a highly efficient and economical way have been implemented as hardware engines instead, resulting in a highly configurable tool box of vision processing units. Analog Devices calls this the “Pipelined Vision Processor” (PVP) which is now part of the new ADSP-BF60x processors. Although implemented on low-power process technology, further innovation was needed to also tackle the dominating power dissipation of modern designs: external memory (DDR) interfaces. Lowest power dissipation is achieved by distribution of processing and by smart utilization of moderate memory bandwidth. In addition, the Blackfin architecture has been enriched by a number of hardware blocks to address functional safety requirements.
Efficient Data Streaming
Many chip architectures receive data from the video sensor and save them frame by frame into the external memories, such as DDR, just to read them back frame by frame (slightly delayed). Multi-core architectures even tend to multiply that data movement for the only benefit of letting the cores identify the region of interests (ROIs) in every frame. The ADSP-BF60x concept avoids such power consuming transport of video data. Still, the full frame is stored in DDR2 memory, but there is no need to read the entire frame back. As soon as it is received by the device, the incoming data is multi-casted to the Pipelined Vision Processor (PVP) which preprocesses the data directly on its way in.
Figure 1: Vision Processing with ADSP-BF60x Processors
As shown in Figure 1, the PVP Camera Pipes can generate up to three intermediate preprocessing results such as edge map, integrals or re-quantification by non-linear thresholds. On top, it can provide status information like histograms. Not a single instruction has to be executed on the Blackfin cores and not a single byte has to be transferred over the DDR bus in order to achieve these results. Additionally, the PVP uses dense data formats when outputting intermediate results and hence, most of the time, the results fit into on-chip L1 and L2 memories.
From here, software running on the Blackfin cores can take it further. The results provided by the PVP Camera Pipes enable software to identify Regions of Interest (ROIs) efficiently. Now, only the ROIs need to be read back from external memory for further analysis. They can be either read by DMA, by the cores or by the PVP Memory Pipe. The latter can further analyze or massage the ROI data, by filtering it or scaling it to the preferred template size. Again, the histogram results come for free. If the dense output of the Memory Pipe is stored into on-chip memories, the Blackfin cores can now locally classify the ROIs and, if needed, verify them against raw data or against respective region in previous frames as stored in DDR2 memory.
Pipelined Vision Processor (PVP) - A Closer Look
The PVP can process up to four data streams, three Camera Pipes and one Memory Pipe, in addition to counting histogram status outputs. As shown in Figure 2 the PVP tool box consists of twelve processing blocks, which are optimized for various vision processing steps.
Typically, the Memory Pipe utilizes one or multiple processing blocks on demand to analyze a series of ROIs and may change operators on the fly. Camera Pipes tend to process the full frame in a parallel manner and incoming data can be multi-casted to multiple processing branches. The user has full flexibility in interconnecting operators to form the pipes. One possible configuration is shown in Figure 3:
Figure 3: Example Configuration for Object and Pedestrian Detection. For full resolution click here.
The traditional Canny-structure can be identified in the figure. It consists of a low-pass filter with Gaussian coefficients, enhanced Sobel filters with a 5x5 matrix, Cartesian-to-Polar conversion and a Non-Maxima Suppression (NMS) stage. Remaining maxima might be passed through a Threshold block to reduce bit resolution or to run-length compress the output for least memory loading. If post-processed by software, say with a Hough-Transform algorithm, line detection and lane keep assist strategies can be derived from this edge map.
If the gradients are forwarded to an Integral block (Histogram of Gradients, HoG) half of the required processing for Pedestrian Detection has already been accomplished. The square of the low-passed pixels may feed the other Integral block, which then can operate in regular (SAT) or diagonal mode (RSAT). It can reduce the frame resolution at the output if high-resolution is not needed, like in the case of shadow detection.
The shown example configuration demonstrates how lane keep assist as well as pedestrian and vehicle detection can be implemented on a single chip in efficient manner, while the MIPS load of the Blackfin cores is still kept at a moderate level. Unused MIPS and the not-yet-used Memory Pipe functionality can be utilized to integrate head lamp control and/or traffic sign detection on top. All of this is enabled by an ADSP-BF60x device at less than 1.3W at 105°C ambient temperature and can therefore be easily installed behind the central mirror in the car, next to the image sensor. In addition, the ADAS Vision processors from Analog Devices support Day- and Night vision systems, capable of analyzing the surroundings of the car as well as driver monitoring systems to detect the state of the driver.
Affordable Driver assistance
At Analog Devices, ADAS developments for Vision and Radar systems are a focus. With the new dedicated ADAS Vision solution offered in form of the Blackfin BF609 and BF608, ADAS Vision technology is now affordable, supporting mass deployments into all levels of vehicles.
About the authors:
Benno Kusstatscher is Systems Engineering Manager, Embedded, Automotive Segment for Analog Devices GmbH.
Peter Voss is Marketing Manager for ADAS Products, Analog Devices GmbH
- No news
- Bosch drives down fuel consumption - in a salami technique
- Automotive microcontroller benchmark takes energy efficiency into account
- Graphics chip recognizes nearby pedestrians and bicycles
- Study investigates V2V communications with motorcycles
- Lithium-ion batteries withstand 10.000 charging cycles
- Continental supplies powertrain for Renault's e-car Zoe
- Time for a new UI programming paradigm
- Audi picks Analog Devices for semiconductor program
- Volvo builds small e-car series for Siemens
- CAN transceiver family supports CAN FD, offers chokeless EMC performance