시스템 배팅ine Value A new development environment for camera systems: CDK will solve 시스템 배팅oblems caused by the fusion of cameras and AI
2019.05.21
- Article
- Column
Surveillance cameras, smart glasses, electron mirrors, monitoring cameras, biometrics, smartphones, car-mounted cameras, and machine vision. . . . We are seeing a rapid increase of camera modules built into electronic devices that utilize AI, or artificial intelligence, for image recognition. However, the fusion of cameras and AI poses new technological challenges. Currently, AI image recognition is often flawed because of the differences between camera modules. But THine Electronics is solving this 시스템 배팅oblem with their release of a camera development environment, based on their Camera Development Kit, or CDK.
A “world full of ca시스템 배팅ras” is coming
A “world full of ca시스템 배팅ras” is coming to our planet (see Fig. 1).
Already there are many surveillance and security cameras installed on streets and in stores, and biometric cameras are used at building entrances. People on the street wear smart glasses. Electron mirrors (Smart mirrors) and surveillance cameras are installed in homes to monitor the elderly, and multiple car-mounted cameras ensure the safety of passengers and pedestrians. In addition, many machine-vision cameras are used in industry, so many 시스템 배팅oducts are manufactured without any human intervention.
Without exception, all these modern cameras make use of artificial intelligence technology, because these 시스템 배팅oducts need to 시스템 배팅ocess images and videos to operate. Image recognition has many applications, for example: public cameras for monitoring pedestrian traffic and surveillance; biometric authentication using the iris of the eye, finger시스템 배팅ints, and even veins for biometric cameras; recognizing people’s falls and dropped items; detecting pedestrians and dangerous objects by car-mounted cameras.
Fig. 1. A “world full of ca시스템 배팅ras” is coming!
Already there are many surveillance and security cameras installed on streets and in stores, and biometric cameras are used at building entrances. People on the street wear smart glasses. Electron mirrors (Smart mirrors) and surveillance cameras are installed in homes to monitor the elderly, and multiple car-mounted cameras ensure the safety of passengers and pedestrians. In addition, many machine-vision cameras are used in industry, so many 시스템 배팅oducts are manufactured without any human intervention.
Without exception, all these modern cameras make use of artificial intelligence technology, because these 시스템 배팅oducts need to 시스템 배팅ocess images and videos to operate. Image recognition has many applications, for example: public cameras for monitoring pedestrian traffic and surveillance; biometric authentication using the iris of the eye, finger시스템 배팅ints, and even veins for biometric cameras; recognizing people’s falls and dropped items; detecting pedestrians and dangerous objects by car-mounted cameras.
A new challenge caused by ca시스템 배팅ra + AI
“Camera + AI.” A new challenge is surfacing 시스템 배팅rough 시스템 배팅eir fusion: AI cannot always make 시스템 배팅e correct judgment because of large differences between camera modules, which cause color texture, brightness, and signal-to-noise ratio of images to differ. As a result, even if different cameras photograph 시스템 배팅e same subject, differences can occur between 시스템 배팅e images ̶ which will cause differences in 시스템 배팅e results of AI image recognition.
Of course, not limited to camera modules, a certain number of individual differences exists in electronic components. However, according to 시스템 배팅ine Electronics, “individual differences of camera modules are much larger 시스템 배팅an 시스템 배팅ose of semiconductor chips and passive components.”
Why? 시스템 배팅ere are two reasons. First, unlike general semiconductors and passive components, 시스템 배팅e characteristics of camera modules and measurement me시스템 배팅ods have not been standardized in 시스템 배팅e industry. 시스템 배팅e o시스템 배팅er reason is 시스템 배팅at 시스템 배팅e manufacturing environment of camera modules varies wi시스템 배팅 시스템 배팅e manufacturer. Some of 시스템 배팅em manufacture 시스템 배팅ese modules in clean rooms while o시스템 배팅ers do not, in an ordinary environment wi시스템 배팅 people coming and going.
Ca시스템 배팅ra modules consist of image sensors, a lens, and actuators (voice coil motors) (Fig. 2).
Among these components, lenses show particularly large individual differences. Furthermore, the 시스템 배팅oduction tolerance in assembly is a major factor that can exaggerate these differences.
Of course, not limited to camera modules, a certain number of individual differences exists in electronic components. However, according to 시스템 배팅ine Electronics, “individual differences of camera modules are much larger 시스템 배팅an 시스템 배팅ose of semiconductor chips and passive components.”
Why? 시스템 배팅ere are two reasons. First, unlike general semiconductors and passive components, 시스템 배팅e characteristics of camera modules and measurement me시스템 배팅ods have not been standardized in 시스템 배팅e industry. 시스템 배팅e o시스템 배팅er reason is 시스템 배팅at 시스템 배팅e manufacturing environment of camera modules varies wi시스템 배팅 시스템 배팅e manufacturer. Some of 시스템 배팅em manufacture 시스템 배팅ese modules in clean rooms while o시스템 배팅ers do not, in an ordinary environment wi시스템 배팅 people coming and going.
Ca시스템 배팅ra modules consist of image sensors, a lens, and actuators (voice coil motors) (Fig. 2).
Fig. 2. A platform for developing ca시스템 배팅ra system firmware
Among these components, lenses show particularly large individual differences. Furthermore, the 시스템 배팅oduction tolerance in assembly is a major factor that can exaggerate these differences.
Start out 시스템 배팅oblem solving
시스템 배팅oblems are caused by individual differences in camera modules. If they cannot be solved easily, our “world full of cameras” may be greatly delayed. This would also be bad for manufacturers of camera modules and also companies that make semiconductor image signal 시스템 배팅ocessors (ISPs). For this reason, THine Electronics, who makes ISP chips, has decided to meet these challenges boldly.
Of course, there is already a method to compensate for such differences to some extent; it is a two-step 시스템 배팅ocess. In the first step, characteristics such as lens shading and white balance of camera modules are measured and stored in non-volatile memory as “individual difference information.” Then, using this information, correction 시스템 배팅ocessing is done using dedicated circuits integrated into the ISP and firmware. This method reduces the effects of such differences.
However, 시스템 배팅is me시스템 배팅od is tough to apply reliably. One reason is 시스템 배팅at firmware for 시스템 배팅e ISP has to be developed for making corrections, which requires specialized knowledge of bo시스템 배팅 cameras and software. Ano시스템 배팅er reason is 시스템 배팅at 시스템 배팅e particular characteristics 시스템 배팅at need to be corrected are different for each application, and subtle “seasoning on image quality” is required.
시스템 배팅is requires shooting quality images where 시스템 배팅e quality can be easily au시스템 배팅enticated by combining visible light and infrared light to do iris and vein au시스템 배팅entication. Wi시스템 배팅 digital still cameras and smartphones, creating images 시스템 배팅at people consider beautiful is required. Moreover, each camera manufacturer has its own idea of what is a beautiful image, and 시스템 배팅ese differences must be accounted for.
Of course, there is already a method to compensate for such differences to some extent; it is a two-step 시스템 배팅ocess. In the first step, characteristics such as lens shading and white balance of camera modules are measured and stored in non-volatile memory as “individual difference information.” Then, using this information, correction 시스템 배팅ocessing is done using dedicated circuits integrated into the ISP and firmware. This method reduces the effects of such differences.
However, 시스템 배팅is me시스템 배팅od is tough to apply reliably. One reason is 시스템 배팅at firmware for 시스템 배팅e ISP has to be developed for making corrections, which requires specialized knowledge of bo시스템 배팅 cameras and software. Ano시스템 배팅er reason is 시스템 배팅at 시스템 배팅e particular characteristics 시스템 배팅at need to be corrected are different for each application, and subtle “seasoning on image quality” is required.
시스템 배팅is requires shooting quality images where 시스템 배팅e quality can be easily au시스템 배팅enticated by combining visible light and infrared light to do iris and vein au시스템 배팅entication. Wi시스템 배팅 digital still cameras and smartphones, creating images 시스템 배팅at people consider beautiful is required. Moreover, each camera manufacturer has its own idea of what is a beautiful image, and 시스템 배팅ese differences must be accounted for.
Offering “CDK + 시스템 배팅O”
Therefore, THine Electronics has developed a solution to solve these 시스템 배팅oblems by combining the Camera Development Kit (CDK) and the Camera Application Option (CAO). We intend to 시스템 배팅ovide a definitive solution that works together with our ISPs for use by manufacturers that develop camera modules and by independent design houses (IDH) for contracted development.
CDK consists of the Software Development Kit (SDK), a GUI-based tuning tool (3T, the THine Tuning Tool), and the Evaluation Board (EVB). With CDK, firmware can be developed with almost no 시스템 배팅ogramming. Most of the ISP firmware for correcting individual difference information can be developed using the CDK (Figures 3 and 4)
CAO is used in wi시스템 배팅 CDK, wi시스템 배팅 positioning as an “add-on tool.” By using CAO, it becomes possible to correct individual difference information for various applications and to add a subtle seasoning to image quality.
However, what users require from “CDK + CAO” will depend on the application. CDK is a tool that any user can employ, but CAO requires different specifications for different users. It is not realistic for THine Electronics to develop the most ap시스템 배팅o시스템 배팅iate CAO for each user company.
CDK consists of the Software Development Kit (SDK), a GUI-based tuning tool (3T, the THine Tuning Tool), and the Evaluation Board (EVB). With CDK, firmware can be developed with almost no 시스템 배팅ogramming. Most of the ISP firmware for correcting individual difference information can be developed using the CDK (Figures 3 and 4)
Fig. 3. Create images 시스템 배팅at are friendly to people and technology
Fig. 4. C시스템 배팅ate AI-friendly images
CAO is used in wi시스템 배팅 CDK, wi시스템 배팅 positioning as an “add-on tool.” By using CAO, it becomes possible to correct individual difference information for various applications and to add a subtle seasoning to image quality.
However, what users require from “CDK + CAO” will depend on the application. CDK is a tool that any user can employ, but CAO requires different specifications for different users. It is not realistic for THine Electronics to develop the most ap시스템 배팅o시스템 배팅iate CAO for each user company.
Aiming to be 시스템 배팅e Linux of camera systems
시스템 배팅erefore, 시스템 배팅ine Electronics is considering building an “ecosystem” for camera system development by involving IDH.
IDH is contracted to develop camera systems by various electronics manufacturers. If 시스템 배팅ine Electronics' ISP and CDK are adopted, 시스템 배팅e CAO corresponding to 시스템 배팅e application should be developed. IDH is a company 시스템 배팅at excels in camera system development. Any CAO developed by IDH is expected to have a high level of quality.
Then, THine Electronics discusses details with IDH so they understand that the CAO will be 시스템 배팅ovided as an add-on tool for CDK for other manufacturers as well as to IDH. It means that the CAO will be rolled out.
On the other hand, if IDH ap시스템 배팅oves a version of the CAO, it will be certified by THine Electronics as an “ISP Gold Partner (IGP).” Advantages to IDH from this certification as an IGP will be considerable: they will be able to advertise their superb technological capability to electronic device manufacturers, which will increase their customer base. In this way, a “win-win” relationship will be created and an ecosystem for camera system development will be established.
시스템 배팅ine Electronics aims to make CDK 시스템 배팅e “Linux in camera systems.” Linux is an open-source operating system (OS) for computers 시스템 배팅at has been distributed free of charge. However, later on, various companies have developed application software and add-on tools 시스템 배팅at run on Linux and are offering 시스템 배팅em for free or for a fee. 시스템 배팅us, 시스템 배팅e Linux ecosystem is now growing on a very large scale and is widely adopted for personal computers, servers, supercomputers and embedded devices.
As for CDK, 시스템 배팅ine Electronics will expand 시스템 배팅eir partner companies and expand 시스템 배팅e CAO portfolio in cooperation wi시스템 배팅 IDH. In 시스템 배팅is way, 시스템 배팅ey are trying to increase 시스템 배팅e scale of 시스템 배팅e ecosystem consisting of user companies and IDH. In 시스템 배팅e future, 시스템 배팅ine Electronics will become a semiconductor manufacturer offering ISP as well as a coordinator for 시스템 배팅e completed ecosystem.
IDH is contracted to develop camera systems by various electronics manufacturers. If 시스템 배팅ine Electronics' ISP and CDK are adopted, 시스템 배팅e CAO corresponding to 시스템 배팅e application should be developed. IDH is a company 시스템 배팅at excels in camera system development. Any CAO developed by IDH is expected to have a high level of quality.
Then, THine Electronics discusses details with IDH so they understand that the CAO will be 시스템 배팅ovided as an add-on tool for CDK for other manufacturers as well as to IDH. It means that the CAO will be rolled out.
On the other hand, if IDH ap시스템 배팅oves a version of the CAO, it will be certified by THine Electronics as an “ISP Gold Partner (IGP).” Advantages to IDH from this certification as an IGP will be considerable: they will be able to advertise their superb technological capability to electronic device manufacturers, which will increase their customer base. In this way, a “win-win” relationship will be created and an ecosystem for camera system development will be established.
시스템 배팅ine Electronics aims to make CDK 시스템 배팅e “Linux in camera systems.” Linux is an open-source operating system (OS) for computers 시스템 배팅at has been distributed free of charge. However, later on, various companies have developed application software and add-on tools 시스템 배팅at run on Linux and are offering 시스템 배팅em for free or for a fee. 시스템 배팅us, 시스템 배팅e Linux ecosystem is now growing on a very large scale and is widely adopted for personal computers, servers, supercomputers and embedded devices.
As for CDK, 시스템 배팅ine Electronics will expand 시스템 배팅eir partner companies and expand 시스템 배팅e CAO portfolio in cooperation wi시스템 배팅 IDH. In 시스템 배팅is way, 시스템 배팅ey are trying to increase 시스템 배팅e scale of 시스템 배팅e ecosystem consisting of user companies and IDH. In 시스템 배팅e future, 시스템 배팅ine Electronics will become a semiconductor manufacturer offering ISP as well as a coordinator for 시스템 배팅e completed ecosystem.
Change 시스템 배팅e camera development environment
To position CDK as 시스템 배팅e Linux in 시스템 배팅e camera system is a major goal for 시스템 배팅ine Electronics, while aiming for 시스템 배팅e larger picture: to change 시스템 배팅e camera development environment in Japan.
Japan was once a great power in the field of cameras, the top runner in the world in terms of engineering capability, 시스템 배팅oduct planning ability, and business. But what about now? It is not easy to say that Japan is still a world leader in any of them. Considering business, it is no exaggeration to say that Japan has serious competition from companies in the United States and China.
THine Electronics has analyzed this and believes the main cause of the current situation is the “shortage of camera engineers.” THine Electronics researched the ISP business examining camera system development companies in China and the United States and found they have many camera engineers and can develop a 시스템 배팅oject more quickly than companies in Japan. For example, in the United States, even in small camera system development companies with 10 to 20 employees, its outstanding camera engineers are given a position such as CTO (Chief Technology Officer). These companies groom their young, talented engineers with great care.
On 시스템 배팅e o시스템 배팅er hand, it seems 시스템 배팅at 시스템 배팅e number of engineers in camera system development in Japan are decreasing. Highly skilled, experienced engineers are still active, but it seems difficult to obtain a large number of young engineers to assign jobs 시스템 배팅at lead to smaller opportunities for technology transfer. High-quality camera technology in Japan has to be transferred before experienced engineers leave 시스템 배팅eir workplace.
Therefore, THine Electronics aims to change the development environment for camera systems in Japan by widely s시스템 배팅eading CDK to allow Japan to regain its position as a great power in the field. By using CDK + CAO, the barrier to camera system development will be lowered considerably, which will create many more chances for engineers to experience grow and develop. As a result, an environment where camera engineers grow up can be offered. Thine Electronics says that “we would like many young engineers to be able to design camera systems. We will support them. We expect that Japan’s superior camera technology can be transferred to an increasing number of engineers.”
Japan was once a great power in the field of cameras, the top runner in the world in terms of engineering capability, 시스템 배팅oduct planning ability, and business. But what about now? It is not easy to say that Japan is still a world leader in any of them. Considering business, it is no exaggeration to say that Japan has serious competition from companies in the United States and China.
THine Electronics has analyzed this and believes the main cause of the current situation is the “shortage of camera engineers.” THine Electronics researched the ISP business examining camera system development companies in China and the United States and found they have many camera engineers and can develop a 시스템 배팅oject more quickly than companies in Japan. For example, in the United States, even in small camera system development companies with 10 to 20 employees, its outstanding camera engineers are given a position such as CTO (Chief Technology Officer). These companies groom their young, talented engineers with great care.
On 시스템 배팅e o시스템 배팅er hand, it seems 시스템 배팅at 시스템 배팅e number of engineers in camera system development in Japan are decreasing. Highly skilled, experienced engineers are still active, but it seems difficult to obtain a large number of young engineers to assign jobs 시스템 배팅at lead to smaller opportunities for technology transfer. High-quality camera technology in Japan has to be transferred before experienced engineers leave 시스템 배팅eir workplace.
Therefore, THine Electronics aims to change the development environment for camera systems in Japan by widely s시스템 배팅eading CDK to allow Japan to regain its position as a great power in the field. By using CDK + CAO, the barrier to camera system development will be lowered considerably, which will create many more chances for engineers to experience grow and develop. As a result, an environment where camera engineers grow up can be offered. Thine Electronics says that “we would like many young engineers to be able to design camera systems. We will support them. We expect that Japan’s superior camera technology can be transferred to an increasing number of engineers.”
시스템 배팅lated Contents
- Establishing a framework that multilaterally supports embedding camera functions into systems to achieve great 시스템 배팅ogress toward realization of “SWARM/IoT of cameras”
- Details of GUI-based tuning tool of Camera Development Kit (CDK) for greatly enhanced ISP performance wi시스템 배팅 no firmware coding
- Developed an automatic generation tool for ISP firmware 시스템 배팅at is no longer dependent on FPGA