Optimised multi-camera systems for dimensional control in factory environments

Abstract

As part of the United Kingdom’s Light Controlled Factory project, University College London aims to develop a large-scale multi-camera system for dimensional control tasks in manufacturing, such as part assembly and tracking. Accuracy requirements in manufacturing are demanding, and improvements in the modelling and analysis of both camera imaging and the measurement environment are essential. A major aspect to improved camera modelling is the use of monochromatic imaging of retro-reflective target points, together with a camera model designed for a particular illumination wavelength. A small-scale system for laboratory testing has been constructed using eight low-cost monochrome cameras with C-mount lenses on a rigid metal framework. Red, green and blue monochromatic light-emitting diode ring illumination has been tested, with a broadband white illumination for comparison. Potentially, accuracy may be further enhanced by the reduction in refraction errors caused by a non-homogeneous factory environment, typically manifest in varying temperatures in the workspace. A refraction modelling tool under development in the parallel European Union LUMINAR project is being used to simulate refraction in order to test methods which may be able to reduce or eliminate this effect in practice.

Keywords

Industrial photogrammetry camera calibration refraction correction manufacturing part assembly

Introduction

The Light Controlled Factory (LCF) project is developing new methods for multi-dimensional measurement and the tracking of assembly machines and parts in shop floor production. Specific objectives of the research programme are as follows:

To investigate assembly alignment and machine position controlled in real time by large volume metrology (LVM) systems to a positional accuracy of between 10 and 250 µm over assemblies in the range of 10–30 m;

To specify and develop innovative techniques for establishing and compensating for the effects of environmental uncertainty on the dimensional fidelity of large assembly tools and parts (5–30 m) and their tolerance stack-up;

To design and implement a novel, ubiquitous seven-dimensional (7D) measurement environment for the entire factory space, to provide graduated positional accuracy from 10 to 500 µm. (Note 7D = 6 degrees of freedom + time variation, for example, in 6D object tracking.)

Given the importance of this precise measurement requirement for high-quality manufacturing, there is great interest in optimising the performance of a multi-camera 7D photogrammetric system for real-time factory operation.¹ The role of the research group at University College London (UCL) in the project is to enhance the accuracy of optical LVM systems by quantifying and modelling optical effects and environmental variations.

The system under evaluation is a network of low-cost cameras for measuring the locations of many thousands of object points marked with high-quality retro-reflective targets. This provides the high-quality imaging and multi-ray real-time triangulation which are key features of high-accuracy photogrammetric systems.

The general research objective of this project is to develop and refine algorithms and techniques to minimise systematic pointing errors in this system. The specific technical contributions towards this aim are as follows:

Use of detailed camera models based on monochromatic imaging;

It is shown that the camera model varies with illumination frequency (colour) and a specific colour + camera model is an optimal combination.

Removal of eccentricity errors due to elliptical imaging of circular targets;

This effect is known but not routinely incorporated into software algorithms. To be effective, the plane of the target must also be established.

Correction of refraction due to variations in the workspace environment;

Photogrammetric models currently assume that imaging rays are straight lines which is not the case when the refractive index varies.

This project will develop corrections using redundant multi-ray intersections as well as the refraction analysis from the parallel LUMINAR project.

Development background

Photogrammetry is the underlying technology behind the research at UCL. For a comprehensive and detailed overview of photogrammetry as applied to close-range and industrial tasks, see Luhmann et al.²

In summary, photogrammetry delivers three-dimensional (3D) measurements using at least two images of features and targets on an object. Positions of the same object point in the multiple images are projected back into object space, where these back-projected image rays intersect at the required 3D coordinate location (see Figure 1(d)). This process of 3D triangulation assumes idealised pinhole camera imaging with straight-line rays in space (Figure 1(c)). However, real cameras (Figure 1(a)) contain complex combinations of lens elements connected via a camera body to an electronic image sensor. Optical imperfections in the system add geometric distortions and aberrations to the image, as illustrated in Figure 1(b). Given a physically stable camera unit, modelling and calibration can minimise the geometric distortions and lead to corrected image measurement data which approximate very closely to a pinhole camera.

Figure 1.

Camera modelling and 3D imaging: (a) real camera (b) image distortion (c) pinhole camera (d) multi-ray intersection.

As a result of this and other enhancements of the photogrammetric process, high-accuracy photogrammetry has been operational for many years, based on constantly improving camera and imaging technologies. An insight into the measurement accuracies achievable in practice can be seen, for example, in the technical reports available from a high-end commercial systems and service provider such as Geodetic Systems, Inc. (GSI).³ As an illustration, GSI photogrammetric measurements of automotive tooling fixtures and of an aero engine yielded accuracies in the range of 10–20 µm.

However, there are still three areas where the UCL team sees further potential to improve photogrammetric accuracy and operation in a factory environment:

Monochrome camera calibration;

Target measurement and identification;

Mitigation of the effects of environmental refraction.

Monochrome camera calibration

The widespread use of colour cameras has highlighted the requirement to adapt camera models for multiple wavelengths of light and so to eliminate the effects of chromatic aberration. Figure 2 shows in simplified form on the left how a convex lens has different focal lengths for blue and red light. The composite image on the right shows how blue and red illumination of the same retro-reflective target generate different image positions. The quality of target image creation is further compounded by the broad spectral sensitivity of silicon charge-coupled device (CCD) and complementary metal oxide semiconductor (CMOS) sensors such that a wide range of wavelengths are simultaneously recorded.

Figure 2.

The need for RGB calibration.

In an earlier paper,⁴ the authors reported on existing colour camera calibration. Digital colour cameras deliver trichromatic images using either red, green and blue (RGB) mosaic (Bayer) filters or separate filtered sensors to partition the sensor wavelength range into suitable spectral bands. One photogrammetric approach has been to model and correct distortions in the three available colour channels and then recombine the images.⁵ Another has been to work through an extended RGB photogrammetric camera model.^6,7 However, these approaches are limited by the broad spectral imaging properties of the sensors with their three-band outputs. This therefore motivates one aspect of this work which is to create an accurate camera model ‘tuned’ to a particular narrow illumination wavelength that is applicable across a wide range of readily available lens designs.

Target measurement and identification

In photogrammetry, targeting and target illumination are critical to achieving high accuracy. It is common to use retro-reflective circular targets under flash or high-intensity illumination to obtain images resembling star fields. In order to achieve the effect, the illumination source must be close to the optical axis and have a coverage of at least the angular view of the camera. A ring light or flash around the lens is very effective for this purpose.⁸ The resulting imagery permits the most accurate target image localisation and, with modifications to the target design, enables also the automated detection of specific targets.

Circular targets are, in general, imaged obliquely such that their imaged shape is elliptical, rather than circular. The centre of the ellipse, the point normally found by an image-processing algorithm, does not necessarily correspond to the centre of the circular target. For larger targets and highly oblique images, this eccentricity error may be significant. The effect is described in several publications.^9–12 It is interesting to note that the use of a spherical target, rather than a flat, circular target, does not solve the problem because a sphere in object space is also imaged as an ellipse.¹² Geometric correction of target centres, while challenging and often made only to a first approximation, has been implemented in systems such as Australis.¹³

A further difficulty with circular targets is that they are ambiguous. Unique identification of at least some targets is required to establish the relative positions and angular orientations of the multiple image locations used to intersect targets, or to track moving targets, in 3D. In an industrial environment, this process, termed correspondence, can be automated using coded targets which have unique geometric or intensity patterns that can be detected automatically by image-processing algorithms and assigned a corresponding unique ID.¹⁴ Although the majority of coded target systems make use of varying patterns and intensities, some systems take advantage of colour images.^7,15

Mitigation of the effects of environmental refraction

The assumption in Figure 1(c) that light rays travel in straight lines is invalid in many circumstances. Environmental changes in a factory environment, in particular, temperature gradients between cold air on the floor and warm air under the roof, cause light rays to be refracted along their path from target point to camera. In the parallel European Union (EU)-funded LUMINAR project,¹⁶ UCL is developing methods to model the effects of this bending of light rays. A simulation analysis is being developed for LUMINAR, and this is now also being used to simulate refraction errors in the LCF project, as discussed in section ‘Refraction simulation’.

Commercial and research multi-camera systems

Because the current work aims to develop a factory-wide, real-time multi-camera positioning and tracking system, state-of-the-art commercially available systems have also been reviewed. Currently, no large-scale commercial systems are available with similar scope to the 7D concept under development here. For manufacturing purposes, typically in the quality control of parts and assembled objects, multi-camera commercial systems do exist but with smaller operating volumes. Some alternative solutions have the potential to capture measurements within a large volume. For example, motion capture and optical probe systems provide similar functionality, while some purpose-built research systems are also worthy of note. However, in all cases, there are significant weaknesses or barriers to providing the functionality and precision required.

Commercial industrial measurement systems

Products from two commercial companies are cited here as examples of industrial multi-camera systems. The Finnish company Mapvision¹⁷ builds real-time photogrammetric systems for inline component inspection. Typically, this applies to car manufacturing, and their multi-camera system, Quality Gate, in different configurations, can measure car chassis and shells of up to 5-m maximum dimension in a dedicated space.

The German company AICON¹⁸ manufactures a wide range of photogrammetric systems such as TubeInspect, a dedicated measurement box which contains multiple cameras for measuring and inspecting tube elements, again aimed at automotive manufacture. A more recent development is 3D Arena which has multiple cameras attached to an overhead frame. A key target audience is motor vehicle designers who can work unrestricted in the relatively large space below the frame. The suggested configuration is 16 cameras covering a 6 × 8 m² working space. A variant configuration is aimed at automated applications and tracks a scanner manipulated by a robot.

Motion capture systems

Other than the systems designed specifically for manufacturing applications, motion capture systems are the most familiar and most widely used commercial multi-camera systems that have a relatively large working volume. With applications ranging across movement analysis, 3D animation, biomechanics and 3D environment design, systems from companies such as OptiTrack,¹⁹ Qualisys²⁰ and Vicon²¹ are well established in the marketplace.

These systems do provide a high level of 7D functionality for both rigid and articulated objects, generally using spherical targets as the signalised points of interest. These may be coupled with user-defined interrelationships between targets to define motion structures, for example, representing the torso, head, arms and legs of human subjects within the field of view of the system. In addition, these systems have been demonstrated to operate successfully throughout a large volume such as a section of a basketball court.

However, the precision and accuracy of target coordinates are typically not first priorities for motion capture systems. Independent testing indicates precision and accuracy of the order of ±0.1–1 mm for this type of system.^22,23 Examples of the use of motion capture systems for industrial and manufacturing purposes are not common, although system manufacturers do target engineering audiences. As an example, an OptiTrack system has been employed to identify fastener locations for assembly verification in the automotive industry.²⁴ Static error levels are reported as ±0.1 mm for spherical targets; however, the performance deteriorates to ±2–3 mm for dynamic tracking.

Optical probe systems

Optical probe systems based on multiple cameras and a targeted probe offer another alternative that provides measurement capability within a well-defined working volume. GSI,³ mentioned earlier in the context of achievable accuracies, makes the V-STARS M and D systems which employ two or more purpose-built GSI cameras to provide 3D coordinate measurement in real time at rates of up to 10 points/s. These systems use interchangeable, wireless-operated, tactile probes with standard coordinate-measuring machine (CMM) tips. A dual-camera V-STARS D system is stated to have an accuracy of the order of (±5 + 5 µm/m), which is 1:160,000 for a 4-m object.

Metronor²⁵ offer the Duo and Quad systems which employ two and four purpose-built Metronor cameras, respectively. ‘Light pen’ wireless probes are available in a range of sizes with standard CMM tips. The accuracy of the system is quoted as (±25 + 17 µm/m) or 1:45,000 for a 4-m object. Like the GSI system, there is potential to add more cameras to extend the working volume to accommodate large objects. GOM,²⁶ AICON¹⁸ and several other optical metrology companies offer equivalent systems with similar capabilities.

For optical probe systems, the cameras are typically mounted on tripods, usually in pairs or in a rigid housing forming a fixed-base stereo system, in order to provide flexibility for a variety of objects to be measured. The measurement applications are restricted by the practicalities of static objects and clear lines of sight, and there is an inevitable trade-off between the capital cost of the equipment and having sufficient cameras to achieve a high reliability for the measured locations. The fundamental difference with these systems is the necessity of a touch probe to identify the measurement points from a few cameras, rather than affixing circular or spherical targets that are imaged by many cameras.

Research systems

Many research systems have been developed for specific applications. The most common rationale is to keep the capital expenditure below the cost of equivalent commercial systems through the use of low-cost cameras and in-house software development. However, some system developments have clearly been leading technologies that were not available in commercial systems at the time of development, or were addressing a niche market that was too narrow or too specialised. Such systems have been used for some time. Mapvision’s commercial systems, for example, are based on research from the late 1980s. At that time, their prototype used four CCD cameras with a modest resolution of 512 × 512 pixels, which could achieve a target coordinate precision of 1:5000 within an area of a few metres.⁴² Subsequent improvements in image measurement precision, combined with more stable, standard-definition CCD cameras and more sophisticated calibration techniques, have realised more favourable target coordinate precisions of up to 1:30,000.²⁷

Another example of a specialised research system is that developed for particle flow velocimetry.²⁸ Four synchronised cameras capture images of neutrally buoyant particles in a turbulent fluid flow, viewed through the glass wall of a test tank. Within a relatively small volume of approximately 200 × 200 × 50 mm³, the system achieves particle location precisions of the order of 0.1 mm. This result is substantially poorer than an in-air test with a calibration fixture, realising precisions of 1:20,000, due to the refraction and attenuation of the glass and water media, as well as the non-uniformity and uneven lighting of the tracer particles.

Much more recently, Chong and Al-Baghdadi²⁹ used multiple, low-cost high-definition (HD) video cameras to characterise human lower limbs and the sole of the foot during muscle movement and walking, respectively. The location precision for planar targets attached to the skin surface is given as ±0.3 mm within a relatively small volume.

It is common in computer vision research to utilise application interfaces built upon open-source libraries, such as OpenCV³⁰ for two-dimensional (2D)/3D tracking of markers and features. Typical applications range from haptics through robotic control to structural deformation monitoring. Bundle adjustment algorithms, including camera calibration incorporated into simultaneous localisation and mapping (SLAM) methods,³¹ are in widespread use for mapping the built environment, for example, using image sets taken from unmanned aerial vehicles (UAVs).³²

All of these commercial and research systems reinforce the potential for a very precise and reliable optical measurement system that can operate effectively across a large volume. In general, the systems employ a calibration object translated and rotated within the field of view of the cameras to determine both the calibration and the relative orientation of the camera. However, based on the available literature, the systems are not attempting to model spectral dependencies or are actively avoiding the use of colour in favour of monochrome images. Correction of errors due to environmental fluctuations is unreported.

Test rig to evaluate multispectral imaging

In order to characterise the environmental sensitivity of the proposed system, the UCL team is investigating the use of very low-cost, low-power, light-emitting diode (LED) sources to illuminate targets at narrow wavebands in the visible and near-infrared spectrum.

Low-cost monochrome cameras with CMOS sensors and C-mount lenses have been successfully applied to a wide variety of metrology tasks. As noted previously, for high-accuracy measurement, such cameras are typically equipped with ring lights to image retro-reflective targets as high-contrast image features. Initial research in the LCF project assessed the effect of wavelength on camera calibration parameters for a C-mount wide-angle lens.⁴ White light from a studio flash was filtered through a series of narrowband transmission filters at intervals of 20 nm throughout the spectrum. The results demonstrated the strong influence of illumination wavelength on principal distance (PD) (distance between the lens perspective centre and the image focal plane) and on radial and tangential lens distortions. The results also revealed possible trends in the parameters for principal point (location of the optical axis on the image focal plane), orthogonality (of image x and y axes) and affinity (differential scale in x and y).

This work takes this further with an eight-camera configuration in a rigid framework mounted on an optical bench. This is a small-scale simulation of a factory configuration. All eight camera bodies are identical, but for comparative investigation, six different C-mount lenses have been used, with focal lengths ranging from 3.8 to 10 mm (fisheye). Also the illumination of the targets is now achieved by a low-cost interchangeable annular LED ring containing 18 discrete LEDs at 20 intervals, attached in front of each lens, rather than by a filtered flash. Four different LED illumination wavelengths have been used to test photogrammetric performance: broadband white light and three narrowband sources of RGB.

This test rig (Figure 3) was constructed in the Advanced Structures Laboratory at UCL, based on a Newport optical table of surface dimensions 1800(L) × 1200(W) mm². The superstructure consists of four upright solid aluminium pillars with a horizontal canopy of Dexion angle steels, all secured firmly together with 9-mm steel bolts. Diagonal struts of solid 40 × 8 mm² bar from the midpoints of the pillars to the canopy sides ensure rigidity of the whole structure. Eight Imaging Development Systems (IDS) uEye miniature CMOS cameras are mounted at the four corners and four midpoints of the sides of the canopy. Each camera is mounted individually onto a Manfrotto ball clamp, so that the cameras can be aligned obliquely to view the working area. The framework outer dimensions are 1500(L) × 1070(W) × 1050(H) mm³. The height of each camera above the surface of the optical table is approximately 850 mm.

Figure 3.

View of eight-camera test rig in laboratory.

Each camera is connected by an Ethernet cable to a network gateway, which also provides power. Synchronisation cables enable all cameras to be fired simultaneously by a common trigger signal driven either by a time signal or by an external event. The 60-mm LED illumination rings are mounted on the front of each camera lens by an adapter ring selected to fit the front diameter (Figure 4). Power for LEDs is provided by a twin-wire ring supply from a 12-V transformer, with CCTV plug/socket pairs to enable the cameras to be detached easily.

Figure 4.

Two views of cameras fitted with C-mount lenses and red LED rings.

Purpose-written software VMSCapture_uEye is used to capture the images. The graphical user interface displays live views from all eight cameras, with a slider control for exposure time, as shown in the screen grab in Figure 5. In this graphic, the images all appear to be upside down, because the cameras are mounted upside down, hanging below the canopy.

Figure 5.

Graphical user interface showing live images from eight cameras.

The software captures eight 12-bit monochrome images simultaneously and saves them as TIFF files (16-bit grey-scale format) of size 2560 × 1920 pixels. The operation can be either in a ‘one-shot’ mode, where one image from each camera is captured as a set, or in ‘triggered free-run’ mode where the image sets are captured continuously at the camera frame rate (typically, 3–4 frames/s). When the external trigger signal is applied, then all cameras are synchronised and the images are captured simultaneously.

Photogrammetric target detection

The analysis technique used to determine image point correspondences in a multi-image network is the long-established photogrammetric bundle adjustment.³³ This simultaneously refines parameters for the perspective camera model, target image locations and selected target coordinate datum with high levels of precision and reliability.³⁴ The network set-up is implemented in a digital close-range photogrammetry system, called the Vision Metrology System (VMS), where the retro-reflective targets, which form the basis of the initial sparse triangulation, have been identified in image sets.

The VMS software, developed over the past 20 years, has been used effectively in numerous industrial applications, where very accurate measurements are required with a discrimination range (also known as proportional accuracy) in the range of 1:100,000³⁵ to 1:200,000.³⁶ The software analyses captured multiple images from an object marked with targets, usually retro-reflective dots or coded markers. This study employs a ‘Manhattan’ model as a 3D test object (Figure 6). This consists of a 550 × 550 mm² aluminium baseplate of thickness 10 mm, onto which are affixed 39 anodised aluminium rods of diameter 8 mm and lengths varying from 20 to 305 mm, all perpendicular to the base. Approximately 100 circular retro-reflective targets of 2.5 mm diameter are distributed over the baseplate and on the top of each rod. The targets form a rigid array of points in a 3D coordinate space.

Figure 6.

Two views of the ‘Manhattan’ test object: (left) under overhead room lights and (right) under flash illumination close to the lens where the increased contrast of the circular targets is evident.

Under flash illumination, the targets are visible in the image from any viewpoint within an incidence angle limit of 50°–60° (Figure 6, right). Depending on the type of retro-reflective film used, the response of the targets is consistent in the range of 0°–50° but rapidly degrades for angles of incidence greater than 55°.⁸ The Manhattan test object is targeted with the ‘gold standard’ 3M 7610 Engineer-Grade retro-reflective film to ensure the maximum visibility of targets in the images. The incidence angle limit and combined effects of perspective distortion impose a restriction on the design of the network geometry. The optimal geometry of the camera locations is defined by intersection angles at the targets of 90° in order to achieve minimum, isotropic precisions in the XYZ coordinates.³⁷ In reality, the network design aims to achieve maximum intersection angles of 50°–60° to ensure that a sufficient retro-reflective response is realised from all targets in the field of view. This also ensures that the effects of perspective and chromatic aberration do not result in extreme elliptical shapes (see Figure 2) that would consequently reduce the precision and accuracy of the target image measurements. The consequence of this compromise is a degradation of the overall precision of the target coordinates and anisotropic precisions that favour XY over the Z coordinates, where Z corresponds to the height of the cameras above the XY plane of the optical table. In general, this compromise does not significantly weaken the fidelity of the extracted camera calibration model, especially if several other network design considerations are maintained.³⁸ Eight machine-readable coded targets are also fixed onto the Manhattan baseplate to facilitate automatic orientation of the image target array using image processing.

Starting with initial estimates of camera and target locations and orientation, VMS performs an iterative bundle adjustment to achieve precise results simultaneously for camera calibration parameters, camera locations and orientations and target coordinates.³³ The more targets there are, and the more views of the object from independent camera positions, the more precise and reliable the results can be. Shortis et al.³⁹ analysed the best performance that could be expected from a range of target image measurement algorithms including ellipse fitting, binary centroiding and Gaussian shape fitting. The results of the study confirmed that errors of the order of 0.1 pixels are typical, and that the results are largely unaffected by quantisation or threshold levels. The use of grey-scale images is likely to perform an order of magnitude better than with binary (black/white) images with greater bit depths supporting better target image quality, particularly in applications where a wide dynamic range is present in the working volume. The key requirements for precise image measurements based on intensity-weighted centroids are a target image diameter in the range of 5–10 pixels, an appropriate threshold level that realises a signal of at least 20 grey levels above the background and an exposure level that does not saturate the sensor photosites at the maximum intensity.⁴⁰

Effect of illumination wavelength

The Manhattan target object was placed at the centre of the optical table, within the field of view of all eight cameras. A total of 12 sets of eight images each were captured, with the test object rotated successively by 90° for four azimuthal angles and tilted at three zenithal angles (0°, 10° and 20° approximately). The images were then analysed by VMS software to determine the camera calibration parameters, including the distortion parameters of each lens. The whole procedure was repeated for LED illumination rings of four different wavelengths: red, green, blue and white.

The spectral power distribution of the LEDs was measured with an Ocean Optics USB2000 +spectrometer at intervals of 0.5 nm (Figure 7) and showed that the peak wavelengths of the RGB LEDs are at 637, 515 and 442 nm, respectively, with bandwidths (full width at half maximum (FWHM)) of 20 nm for both R and B and 32 nm for G. The white LED has a broad bimodal curve with a small peak at 442 nm (blue) and a broader peak with maximum at 546 nm corresponding to the phosphor emission (yellow green).

Figure 7.

Normalised spectral power distributions of the four LED sources.

In the images analysed by VMS, the mean precision achieved in both image space and object space may be compared for the four LED sources (Table 1). Given that the sensor photosite dimension in the cameras is 2.2 µm, the Root Mean Square (RMS) image residual of 0.22 µm represents one-tenth of a pixel. Given that the mean distance from all cameras to all targets was 970 mm, the object precision of 4.4 µm (at k = 1) represents an overall system-discriminating power of approximately 1:220,000. All three narrowband LED sources yielded mean object precisions better than for white light, with an improvement of 10% in the case of red and 5% for green and blue, indicating better optical performance of the lenses.

Table 1.

Root Mean Square (RMS) image residuals and object-space target coordinate precisions at k = 1.

LED source	Red	Green	Blue	White
Image (pixels)	0.25	0.22	0.22	0.23
Target (µm)	4.25	4.45	4.42	4.66

LED: light-emitting diode.

It should be noted that the network geometries for the four cases were virtually identical, but there were some minor differences which may have contributed to the variations observed in the image and object space precisions. However, all four photogrammetric networks contained very high levels of redundant information, ensuring that the results have commensurately high levels of reliability.

The radial distortion profiles of the lenses, based on the fitted VMS lens distortion models for conventional and fisheye lenses (Figure 8), show that the geometric distortion for green light was very similar to white, whereas under red light, the magnitude of the radial distortion was greater and under blue light, it was less. The effect was largely independent of radius, as shown by the ratios of the three colour bands relative to white (Figure 9). For this lens, a high-quality Schneider 8 mm, the distortion under red illumination was 5% greater and under blue illumination was 7% less than under green. The ratios differed for the other lenses, but the trends were similar. The effect can be explained by lateral chromatic aberration in the optical path, consistent with the results obtained in the previous multispectral study.⁴

Figure 8.

Radial distortion profiles for a Schneider 8-mm lens with all four LED sources.

Figure 9.

Radial distortion ratios: red, green, blue versus white.

Other properties of lenses were also dependent upon wavelength. In the previous study, PD followed a clear ‘parabolic’ trend with a minimum at around 500 nm and rising for both shorter and longer wavelengths. This was to be expected for an ‘achromat’ lens optical design corrected over the range of visible wavelengths to minimise chromatic aberration. In this study, PD was calculated as one of the 10 parameters of the VMS lens distortion model, for each of the eight lenses under the four LED wavelengths (Table 2).

Table 2.

Principal distance (PD) for eight lenses under four LED sources.

	Lens	PD (mm)
1	Schneider 8 mm	8.2535	8.2272	8.2146	8.2295
2	Koyo 5 mm	5.0187	5.0096	5.0129	5.0114
3	Kern 10 mm	10.2033	10.1769	10.1692	10.1816
4	Fujinon 9 mm	8.9149	8.8831	8.8666	8.8857
5	Schneider 4.8 mm	4.9742	4.9678	4.9718	4.9714
6	Koyo 5 mm	5.0224	5.0138	5.0191	5.0162
7	Computar 3.8 mm	3.8363	3.8339	3.8418	3.8348
8	Fujinon 9 mm	9.0007	8.9693	8.9520	8.9680

The ratios of the PD values for the RGB sources relative to the white source are plotted in Figure 10. There is good consistency between the repeated instances of the Koyo 5-mm lens (numbers 2 and 6) and the Fujinon 9-mm lens (numbers 4 and 8). The behaviour for the green band is stable relative to white light, with the ratio close to 1, and in all cases except the fisheye (Computar 3.8 mm, number 7), the PD for blue is less than the PD for red. But it is evident that different lens designs cause them to behave in different ways, and a general trend cannot readily be predicted. The effect of a broad range of wavelengths is to spread the range of PD by about 0.5% and hence to defocus the image under white light and reduce the precision of the photogrammetric system.

Figure 10.

Ratios of principal distance (PD) for red, green and blue versus white LEDs.

Refraction simulation

In the LUMINAR project,¹⁶ the UCL team is analysing the effect of refraction in a factory environment and evaluating mechanisms to reduce or eliminate the consequent image location errors.

Optical methods such as photogrammetry assume that light travels in straight lines. However, when the working environment is not homogeneous, typically when there are temperature variations present, refraction causes the rays to be deflected away from the ideal straight line. Figure 11 illustrates the effect using the example of a vertically increasing thermal gradient, a situation which can commonly arise in an enclosed, factory environment.

Figure 11.

Bending of light rays due to a vertically increasing thermal gradient.

LUMINAR considers the effect of ray bending at a single instrument such as a camera or laser tracker. LCF deals with multiple measuring locations, and the LCF camera networks generate multiple rays from different directions into the same target locations. This highly redundant 3D measurement configuration, a key feature in enhancing photogrammetric precision and reliability, potentially also offers a way of detecting errors due to refraction. This, in turn, may lead to ways of correcting the induced errors and enhancing the accuracy of the results.

Analytical tools under development in LUMINAR for simulating refraction can therefore be used to simulate two sets of multiple camera images in an LCF configuration. One set represents images free of refraction errors, while the other set includes the apparent image shifts due to ray bending caused by a particular, defined, inhomogeneous environment. Figure 11 illustrates a simple example of such an environment, although more complex thermal distributions can also be simulated. In addition to simulating image sets with and without refraction effects, random measurement error can be introduced into the simulated image measurements in order to test any modelling technique for robustness.

Figure 12 shows a screenshot of a simulation in Mathcad. This is a view generated by a camera with focal length of 5 mm and located at one of the upper corners of an example measuring box, which is 4.5m high, 6m long and 6m wide. There is a uniform vertical thermal gradient of approximately 2°C, and target points have been simulated on the base of the box, arranged to form simple patterns to aid visual identification (bottom right). On the left of Figure 12 are two superimposed simulated images. The black dots are the target images without refraction (reference set), and the red dots are with refraction (test set). A scale factor (in this case, 5000) has been applied to the difference between refracted and un-refracted image positions in order to make them visible. In this example, the environmental model is defined by horizontal layers of air at different temperatures, increasing linearly with height. This causes an apparent upward vertical shift of the targets, so that the displacement is similar to the perspective effect caused, for example, when photographing buildings from above. Here, the roof corners are displaced from their corresponding ground positions in a similar way. For the configuration shown, the maximum and minimum apparent target displacements in object space are 32.1 and 8.6 µm, respectively. The average apparent displacement is 17.8 µm.

Figure 12.

Simulated image in a Mathcad analysis of targets with and without refraction present.

The current status of this aspect of the work is that refracted images can be simulated and target image measurements can be generated for subsequent bundle adjustment within VMS. However, the refraction model is still being experimentally validated in the LUMINAR project. In addition, the LCF work will need to consider new analysis methods, potentially with enhancements to VMS, in order to mitigate the effects of refraction in a factory environment.

Conclusion

This article has presented the status of the LCF project and the relevant aspects of the LUMINAR project, both based at UCL. The preliminary results produced by calibrations of a test rig of eight IDS uEye CMOS cameras with a variety of lenses have been analysed to demonstrate the impact of different wavelength bands. The results indicate that there are significant effects on the estimated camera calibration parameters; however, the measures of precision in both image and object space are relatively unaffected with sub-pixel measurement precisions between 0.22 (1/10th pixel) and 0.25 µm being achieved. Scaling up the working volume and increasing the number of cameras will enable further analysis of the measures of precision. Experimental work in increasingly large volumes will support a full evaluation of accuracy and the development of a project demonstrator.

In addition, some early results from the LUMINAR project have presented the initial physical modelling necessary to compensate for variations in the factory environment. The case of vertical layers of increasing atmospheric temperature produces small but systematic effects on the image space locations of targets. Detection and compensation of these errors will be challenging, since they are comparable in magnitude to the sub-pixel measurement capability of the camera technologies. However, more complex atmospheric models, combined with high levels of redundancy within the photogrammetric network, conferred by the relatively low cost of adding additional cameras, have the potential to increase the effectiveness of the modelling and the applicability to factory floor requirements.

Further work in the LCF research project at UCL will include the following:

Selection of a lens with an acceptable cost/performance trade-off that can be optionally fitted to all cameras to give uniform coverage of the specified measurement volume;

Verification of the measurement accuracy of the eight-camera array using one or more calibrated scale bars in various orientations within the common intersection volume seen by all cameras. For this validation of the system, the VDI/VDE 2634 performance specification will be used;⁴¹

Scaling up the system by doubling each dimension to a cuboid of approximately 4 × 2 × 2 m³ and adding further eight cameras;

Investigating options to reduce the effects of refraction, using both simulations and measurements at differing temperatures.

Footnotes

Authors’ note

This paper is a revised and extended version of the paper presented at the 2015 conference on Sustainable Design and Manufacture.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This investigation was supported by EPSRC Grant EP/K018124/1 ‘The Light Controlled Factory’.

References

Peggs

Maropoulos

Hughes

. Recent developments in large-scale dimensional metrology. Proc IMechE, Part B: J Engineering Manufacture 2009; 223(6): 571–595.

Luhmann

Robson

Kyle

. Close-Range photogrammetry and 3D imaging. Berlin: De Gruyter, 2014.

Geodetic Systems, Inc. (GSI), http://www.geodetic.com (accessed October 2015).

Robson

MacDonald

Kyle

. Multi-spectral calibration to enhance the metrology performance of Cmount camera systems. In: Proceedings of the ISPRS conference on close-range imaging, ranging and applications, Riva del Garda, Italy, 23–25 June 2014, Volume XL-5, pp.517–521. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

Mallon

Whelan

Calibration and removal of lateral chromatic aberration in images. Pattern Recogn Lett 2007; 28(1): 125–135.

Luhmann

Hasstedt

Tecklenburg

Modelling of chromatic aberration for high precision photogrammetry. ISPRS Arch 2006; 36(5): 173–178.

Cronk

Fraser

Hanley

Automated metric calibration of colour digital cameras. Photogramm Rec 2006; 21(116): 355–372.

Burgess

Shortis

Scott

Photographic assessment of retroreflective film properties. ISPRS J Photogramm 2011; 66: 743–750.

Dold

Influence of large targets on the results of photogrammetric bundle adjustment. Int Arch Photogramm Remote Sens 1996; 31(Part B5): 119–123.

10.

Ahn

Warnecke

HJ.

Systematic geometric image measurement errors of circular object targets: mathematical formulation and correction. Photogramm Rec 1999; 16(93): 485–502.

11.

Otepka

Fraser

. Accuracy enhancement of vision metrology through automatic target plane determination. In: Proceedings of the ISPRS congress, Istanbul, Turkey, 12–23 July 2004, vol. XXXV, part B, pp.873–879. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

12.

Luhmann

Eccentricity in images of circular and spherical targets and its impact on spatial intersection. Photogramm Rec 2014; 29(148): 417–433.

13.

Australis, http://www.photometrix.com.au/australis/ (accessed January 2016).

14.

Shortis

Seager

JW.

A practical target recognition system for close range photogrammetry. Photogramm Rec 2014; 29(147): 337–355.

15.

Moriyama

Kochi

Yamada

. Automatic target-identification with the color-coded-targets. Int Arch Photogramm Remote Sens 2008; 37(B5): 39–44.

16.

NPL. LUMINAR project, http://projects.npl.co.uk/luminar/ (accessed October 2014).

17.

Mapvision, http://www.mapvision.fi (accessed October 2015).

18.

AICON, http://www.aicon3d.de (accessed October 2015).

19.

OptiTrack, http://www.optitrack.com (accessed October 2015).

20.

Qualisys, http://www.qualisys.com (accessed October 2015).

21.

Vicon, http://www.vicon.com (accessed October 2015).

22.

Carse

Meadows

Bower

. Affordable clinical gait analysis: an assessment of the marker tracking accuracy of a new low-cost optical 3D motion analysis system. Physiotherapy 2013; 99(4): 347–351.

23.

Windolf

Götzen

Morlock

Systematic accuracy and precision analysis of video motion capturing systems – exemplified on the Vicon 460 system. J Biomech 2008; 41(12): 2776–2780.

24.

Rusli

Luscher

Fastener identification and assembly verification via IR tracking. Assemb Autom 2012; 32(3): 262–275.

25.

Metronor, http://www.metronor.com (accessed October 2015).

26.

GOM, http://www.gom.com (accessed October 2015).

27.

Clarke

Cooper

MAR

Chen

. Automated 3D measurement using multiple CCD camera views. Photogramm Rec 1995; 15(85): 27–42.

28.

Maas

Grün

Digital photogrammetric techniques for high-resolution 3-D flow velocity measurements. Opt Eng 1995; 34(7): 1970–1976.

29.

Chong

Al-Baghdadi

JAA

. Accurate videogrammetric data for human limb movement research. In: Proceedings of the IEEE international conference on signal and image processing applications (ICSIPA), Melaka, Malaysia, 8–10 October 2013, pp.156–160. New York: IEEE.

30.

Salas-Moreno

Newcombe

Strasdat

. SLAM++: simultaneous localisation and mapping at the level of objects. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Portland, OR, 23–28 June 2013.

31.

Pradeep

Konolige

Berger

Calibrating a multi-arm multi-sensor robot: a bundle adjustment approach. In: Khatib

Kumar

Sukhatme

(eds) Experimental robotics (Springer tracts in advanced robotics, vol. 79). Berlin, Heidelberg: Springer, 2011, pp.211–225.

32.

Bemis

Micklethwaite

Turner

. Ground-based and UAV-based photogrammetry: a multi-scale, high-resolution mapping tool for structural geology and paleoseismology. J Struct Geol 2014; 69(A): 163–178.

33.

Granshaw

SE.

Bundle adjustment methods in engineering photogrammetry. Photogramm Rec 1980; 10(56): 181–207.

34.

Luhmann

. 3D imaging: how to achieve highest accuracy. Proc SPIE 2011; 8085: 808502.

35.

Shortis

Ogleby

Robson

. Calibration modelling and stability testing for the Kodak DC20 series digital still camera. In: Proceedings of the SPIE conference on videometrics and optical methods for 3D shape measurement, San Jose, CA, 20 January 2001, vol. 4309, pp.148–153. Bellingham, WA: SPIE.

36.

Shortis

Burgess

Photogrammetric monitoring of the construction of a solar energy dish concentrator. Photogramm Eng Rem S 2012; 78(5): 519–527.

37.

Fraser

CS.

Network design. In: Atkinson

(ed.) Close range photogrammetry and machine vision. Caithness: Whittles Publishing, 1996, pp.256–281.

38.

Kenefick

Gyer

Harp

BF.

Analytical self calibration. Photogramm Eng Rem S 1972; 38(11): 1117–1126.

39.

Shortis

Clarke

Robson

. Practical testing of the precision and accuracy of target image centering algorithms. In: Proceedings of the SPIE conference on videometrics IV, Philadelphia, PA, 22 October 1995, vol. 2598, pp.65–76. Bellingham, WA: SPIE.

40.

Trinder

JC.

Precision of digital target location. Photogramm Eng Rem S 1989; 55(6): 883–886.

41.

VDI/VDE. Optical 3D measuring systems – imaging systems with point-by-point probing. In: VDI/VDE 2634, part 1, 2002, pp. 1–10. Berlin: Beuth Verlag, Verein Deutscher Ingenieure.

42.

Haggrén

Leikas

Mapvision: the photogrammetric machine vision system. Photogramm Eng Rem S 1987; 53(8): 1103–1108.