Technical field

[0001] The invention generally relates to a sensor-based monitoring and/or surveillance system and the use of such system e.g. in an automotive vehicle interior or exterior sensing system.

Background of the Invention

[0002] Systems based on sensor input experience fast-growing demands in various fields. For instance, in the automotive field they constitute the backbone of almost all Advanced Driver-Assistance Systems (ADAS) as these monitor an exterior environment or the interior of a vehicle and its occupants for providing improved safety by facilitating an optimized reaction of a driver of a vehicle with appropriate warnings or even by automatically taking over control of the vehicle, for instance in collision avoidance systems.

[0003] In this function, such systems are requested to perform tasks of increasing complexity. For example, they should be capable to anticipate potential risks that might occur in complex traffic scenarios within the next few seconds. In conventional ADAS, usually an electronic processing unit such as a central processing unit (CPU) is employed for executing a program code of a software module that has been manually designed for controlling an automatic execution of a monitoring method.

[0004] By way of example, patent application publication US 2014/0139670 A1 describes a system and method directed to augmenting advanced driver assistance systems (ADAS) features of a vehicle with image processing support in an on-board vehicle platform. Images may be received from one or more image sensors associated with an ADAS of a vehicle. The received images may be processed. An action is determined based upon, at least in part, the processed images. A message is transmitted to an ADAS controller responsive to the determination. To that end, the vehicle may include one or more processors, networking interfaces, and other computing devices that may enable it to capture image data, process the image data, and augment ADAS features of the vehicle with image processing support in the on-board vehicle platform. A computing system may include single-feature fixed-function devices such as an ADAS image system on chip (SoC). The SoC may include a microcontroller, microprocessor, or digital signal processor core(s). In some embodiments, SoCs may include more than one processor core. SoCs may further include blocks, which may include a ROM, RAM, Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory, and/or any other type of non-volatile memory.

Object of the invention

[0005] It is therefore an object of the invention to provide a sensor-based monitoring or surveillance system with improved capability regarding the execution of complex tasks, particularly for automotive and medical monitoring and for surveillance applications.

General Description of the Invention

[0006] The complexity of tasks to be performed by an ADAS more and more tends to grow beyond the practicability of manually designed algorithms. Within the scope of the invention, it is therefore proposed to exploit the capabilities of artificial intelligence (Al) systems or modules in general and more particularly artificial neural networks, respectively, in a way that only the basic architecture of the system has to be defined and algorithms are learned based on received input data. In particular, an Al system that is targeted to carry out any of the aforementioned complex tasks should be equipped with a data memory for taking into account information/data that date back up to a predetermined time period.

[0007] In general, mechanisms for providing an artificial neural network with read- write access to an external memory have been described in the research article by Graves, Alex, et al.: "Hybrid computing using a neural network with dynamic external memory" , Nature 538, 471 -476, (27 October 2016), which shall hereby be incorporated by reference in its entirety with effect for those jurisdictions permitting incorporation by reference.

[0008] In the article, artificial neural networks are described to be remarkably adept at sensory processing, sequence learning and reinforcement learning, but are limited in their ability to represent variables and data structures and to store data over long timescales, owing to the lack of an external memory. A machine learning model is introduced called a differentiable neural computer (DNC), which consists of a neural network that can read from and write to an external memory matrix, analogous to the random access memory in a conventional computer. Like a conventional computer, it can use its memory to represent and manipulate complex data structures, but, like a neural network, it can learn to do so from data. When trained with supervised learning, a DNC can successfully answer synthetic questions designed to emulate reasoning and inference problems in natural language. It can learn tasks such as finding the shortest path between specified points and inferring the missing links in randomly generated graphs, and then generalize these tasks to specific graphs such as transport networks and family trees. When trained with reinforcement learning, a DNC can complete a moving blocks puzzle in which changing goals are specified by sequences of symbols. Taken together, DNCs are held to have the capacity to solve complex, structured tasks that are inaccessible to neural networks without external read-write memory.

[0009] In one aspect of the present invention, the object is achieved by a sensor- based artificial intelligence monitoring and/or surveillance system, which includes an artificial intelligence module, an external memory unit and at least one sensor system.

[0010] The artificial intelligence module may comprise any suitable Al system such as e.g. artificial neural networks, support vector machines (SVM) or the like. In a preferred embodiment, the artificial intelligence module comprises e.g. one or more artificial neural networks comprising a plurality of interconnected artificial neurons. The artificial intelligence module has at least one input side and at least one output side. The artificial intelligence module also has selective reading and selective writing access to the external memory unit via a read/write interface having a trainable implementation by any suitable training method. The read/write interface having a trainable implementation could for instance be a differentiable interface so that the artificial intelligence module has selective reading and selective writing access to the external memory unit by using mathematically differentiable attention mechanisms. The at least one sensor system is connected to the input side of the artificial intelligence module, and each sensor system comprises at least one sensor device. [0011] The artificial intelligence module is configured to provide, at the output side, an output signal that represents a specific situation with regard to at least one object to be monitored or surveyed by applying at least one trained task to signals that have been received from the at least one sensor system and/or any other data derived thereof, such as e.g. intermediate processing results derived from sensor signals from preceding time frames, that have been stored in the external memory unit and that date back up to a predetermined time period.

[0012] The phrase “being configured to”, as used in this application, shall in particular be understood as being specifically programmed, laid out, furnished or arranged.

[0013] In this way, a sensor-based monitoring and/or surveillance system can be provided with an improved capability of executing complex tasks, as tasks to be executed by the system are no longer given by manually designed computer algorithms but rather can be learned based on data represented by received sensor signals. The proposed monitoring and/or surveillance system can have access to live sensor signal inputs as well as to intermediate processing results from preceding time frames, or which date back up to a longer time. By that, such system can have the capability to automatically learn tasks that require taking into account data from the past by managing its own memory.

[0014] As is well known in the field of artificial neural networks, each artificial neurons of the plurality of interconnected artificial neurons (also called nodes) can transmit a signal to another artificial neuron connected to it, and the received signal can further be processed and transmitted to the next artificial neuron. The output of each artificial neuron may be calculated using a non-linear function of the sum of its inputs. In a learning process, weights of the non-linear function usually are being adjusted. A complex task may be learned by determining a set of weights for the artificial neurons such that the output signal of the artificial neural network is close to a desired output signal, which is performed when the artificial neural network is trained. Multiple methods for training an artificial neural network, such as the backpropagation algorithm, are known in the art. The prefered differentiable attention mechanisms allow for the artificial neural network to learn by applying some variant of the stochastic gradient descent method or any other type of a continuous optimizer to a loss function. [0015] The proposed sensor-based monitoring and/or surveillance system may be beneficially employed in the automotive field, in the field of medical diagnosis devices, and in the field of smartphones and drones. It should however be noted that the proposed sensor-based monitoring and/or surveillance system is not limited to these fields.

[0016] In preferred embodiments, the artificial intelligence module is formed by a deep neural network (DNN), such as a recurrent neural network (RNN) or a feedforward neural network FNN). A DNN is an artificial neural network with multiple hidden layers of artificial neurons between the input and the output. DNNs are known to be able to model complex non-linear relationships. An RNN is a subset of DNN and is an artificial neural network whose connections between nodes form a directed graph along a sequence. RNNs can show dynamic temporal behavior for a time sequence. A FNN also is a subset of DNN and is an artificial neural network wherein connections between the nodes do not form a cycle. Both subset types of artificial neural networks are beneficially employable in the proposed sensor-based monitoring and/or surveillance system.

[0017] Preferably, the predetermined time period has a length between 0.1 and 10 seconds, which is especially suitable for monitoring and/or surveillance systems that are to be employed in automotive applications, particularly in automotive exterior sensing. In this application, an individual traffic user might be outside a field of view of a sensor system, for instance a radar sensor system, or covered by another traffic user for a duration shorter than the predetermined time period.

[0018] In preferred embodiments, the predetermined time period has a length of up to 15 minutes. This is especially suitable for monitoring and/or surveillance systems that are to be employed in surveillance applications, in which it is desirable to be able to trace the path of a specific individual or object.

[0019] Preferably, the at least one sensor system comprises at least one out of an optical camera, a radar sensor system and a Lidar (light detection and ranging) device. By that, sensor signals can be provided that allow to detect characteristic features of an object to be monitored or surveilled in a variety of ways, depending on the specific application. [0020] In preferred embodiments, in which the monitoring and/or surveillance system includes a radar sensor system, the radar sensor system is configured to be operated at a radar carrier frequency that lies in a frequency range between 20 GHz and 90 GHz. In this frequency range, radar components are readily available. Further, a wavelength of the radar carrier frequency in this frequency range is most suitable for detecting objects that potentially occur in regular traffic conditions, such as, but not limited to, oil spill, large inanimate obstacles, black ice, snow, animals, pedestrians, cyclists and other vehicles. At the upper end of the frequency range, a wavelength of the radar carrier frequency is most suitable for detecting details of vehicle occupants, such as, but not limited to, a breathing activity of a driver.

[0021] In another aspect of the invention, it is proposed to use the disclosed sensor-based monitoring and/or surveillance system, including at least one out of an optical camera, a radar sensor system, in particular a radar sensor system configured to be operated at a radar carrier frequency that lies in a frequency range between 20 GHz and 90 GHz, and a Lidar device in an automotive vehicle exterior sensing system.

[0022] The term “vehicle”, as used in this application, shall particularly be understood to encompass passenger cars, trucks, semi-trailer tractors and buses.

[0023] In yet another aspect of the invention, it is proposed to use the disclosed sensor-based monitoring and/or surveillance system, including at least one out of an optical camera and a radar sensor system, in particular a radar sensor system configured to be operated at a radar carrier frequency that lies in a frequency range between 20 GHz and 90 GHz, in an automotive interior sensing system. Such monitoring and/or surveillance system can beneficially be employed for, without being limited to, a detection of left-behind pets and/or children, vital sign monitoring, vehicle seat occupancy detection for seat belt reminder (SBR) systems, and anti-theft alarm.

[0024] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

[0025] It shall be pointed out that the features and measures detailed individually in the preceding description can be combined with one another in any technically meaningful manner and show further embodiments of the invention. The description characterizes and specifies the invention in particular in connection with the figures.

Brief Description of the Drawings

[0026] Further details and advantages of the present invention will be apparent from the following detailed description of not limiting embodiments with reference to the attached drawing, wherein:

Fig. 1 illustrates a schematic diagram of an embodiment of the sensor-based monitoring and/or surveillance system in accordance with the invention,

Fig. 2 illustrates a schematic diagram of an alternative embodiment of the sensor-based monitoring and/or surveillance system in accordance with the invention, and

Fig. 3 illustrates a schematic diagram of another alternative embodiment of the sensor-based monitoring and/or surveillance system in accordance with the invention.

Description of Preferred Embodiments

[0027] Fig. 1 illustrates a schematic diagram of an embodiment of the sensor- based monitoring and/or surveillance system in accordance with the invention. The monitoring and/or surveillance system 10 is configured and intended to be used in an automotive vehicle exterior sensing system. It includes an artificial intelligence module (Al module) 12 in the form of an artificial neural network 12 that is formed as a deep neural network (DNN) with a plurality of interconnected artificial neurons establishing an input layer, an output layer and a plurality of hidden layers, wherein the input layer forms part of an input side 14 and the output layer forms part of an output side 16 of the artificial neural network 12.

[0028] The sensor-based monitoring and/or surveillance system 10 further includes an external memory unit 18 that in this specific embodiment is formed by a real-valued memory matrix. The artificial neural network 12 has selective reading and selective writing access to the external memory unit 18 by using attention mechanisms known in the art as read head 20 and write head 22, respectively, which are mathematically differentiable. [0029] The external memory unit 18 and the read 20 and write heads 22 are integrated such that usage of the external memory unit 18 is trainable during a task training process. Hence, the monitoring and/or surveillance system 10 can be trained by applying some variant of the stochastic gradient descent method known in the art, or any other type of a continuous optimizer that appears to be suitable to those skilled in the art.

[0030] Connected to the input side 14 of the artificial neural network 12, the monitoring and/or surveillance system 10 includes a radar sensor system 24 comprising a plurality of radar sensor devices formed as radar transceivers (not shown). In this specific embodiment, the radar sensor system 24 is formed as a phase-modulated continuous wave (PMCW) radar system configured to be operated at a radar carrier frequency that lies in a frequency regime between 20 GHz and 90 GHz, for example at a radar carrier frequency of 22 GHz.

[0031] Optionally or additionally, the sensor-based monitoring and/or surveillance system 10 may include an optical camera 26, whose signal output line 28 can be connected to the input side 14 of the artificial neural network 12, and that may be fixedly or movably connected to a chassis of the vehicle.

[0032] Further optionally or additionally, the sensor-based monitoring and/or surveillance system 10 may include a Lidar device 30, whose signal output line 32 can be connected to the input side 14 of the artificial neural network 12, and that may be fixedly or movably connected to a chassis of the vehicle.

[0033] The artificial neural network 12 is configured to provide, at the output side 16, an output signal 34 that represents a specific situation with regard to at least one object to be monitored by applying at least one trained task to signals that have been received from the radar sensor system 24 and, if applicable, from the optical camera 26 and/or the Lidar device 30, have been stored in the external memory unit 18 and that date back up to a predetermined time period. The predetermined time period has a length between 0.1 and 10 seconds, and in this specific embodiment amounts to about eight seconds.

[0034] The purpose of this specific embodiment of the disclosed sensor-based monitoring and/or surveillance system 10 is to detect objects in the near and mid- range environment of the vehicle, such as, but not limited to, other vehicles, pedestrians, traffic signs/lights, buildings etc., and to make predictions of potential objects that might intersect the route of the vehicle to which the sensor-based monitoring and/or surveillance system 10 is attached, within a certain period in the future. The monitoring and/or surveillance system 10 receives data from the radar sensor system 24 and, if applicable, from the optical camera 26 and the Lidar device 30, based on which it is capable to perform a very reliable object detection and classification. In a typical traffic scenario, some dynamic objects may be detectable only in limited time slots. Nonetheless, the monitoring and/or surveillance system 10 is capable of taking into account such objects, even if they are outside a field of view of every connected sensor system 24, 26, 30 for several time frames. For instance, a cyclist that has been detected two seconds ago but that has been covered by a bus since that time might still be relevant. By applying a trained task to the received sensor signals that have been stored in the external memory unit 18 and that date back up to the predetermined time period of about eight seconds, the monitoring and/or surveillance system 10 is capable to provide an output signal 34 representing a specific situation with regard to the cyclist.

[0035] Fig. 2 illustrates a schematic diagram of an alternative embodiment of the sensor-based monitoring and/or surveillance system 40 in accordance with the invention. In order to avoid unnecessary repetitions, only differences with respect to the first embodiment will be described. For features in Fig. 2 that are not described, reference is made to the description of the first embodiment.

[0036] The alternative embodiment of the sensor-based monitoring and/or surveillance system 40 includes an artificial neural network 12 and a sensor system formed by an optical camera 26, whose signal output line 28 is connected to an input side 14 of the artificial neural network 12. The artificial neural network 12 is of the same type as in the first embodiment, but may alternatively be formed as a recurrent neural network (RNN).

[0037] The artificial neural network 12 is configured to provide, at the output side 16, an output signal 34 that represents a specific situation with regard to at least one object to be monitored or surveilled by applying at least one trained task to signals that have been received from the optical camera 26, have been stored in the external memory unit 42 and that date back up to a predetermined time period. The predetermined time period has a length of up to 15 minutes, and in this specific embodiment amounts to about ten minutes.

[0038] This specific embodiment of the disclosed sensor-based monitoring and/or surveillance system 40 is intended to be used for surveillance purposes at large public buildings, like for instance airports. The monitoring and/or surveillance system 40 is configured to monitor people and to predict any potential criminal or terroristic activity. In order to assess suspicious persons, the artificial neural network 12 takes into account data acquired in advance that date back up to about 10 min. A typical monitoring task could be to determine, for instance, to whom a person has communicated or whether any observed walking route is in line with common behavior.

[0039] Fig. 3 illustrates a schematic diagram of another alternative embodiment of the sensor-based monitoring and/or surveillance system 50 in accordance with the invention. In order to avoid unnecessary repetitions, only differences with respect to the first embodiment will be described. For features in Fig. 3 that are not described, reference is made to the description of the first embodiment.

[0040] The alternative embodiment of the sensor-based monitoring and/or surveillance system 50 pursuant to Fig. 3 includes an artificial neural network 12, a sensor system formed by an optical camera 26 and another sensor system formed by a radar sensor system 54, whose signal outputs are connected to an input side 14 of the artificial neural network 12. The artificial neural network 12 is of the same type as in the first embodiment, but may alternatively be formed as a recurrent neural network (RNN). The radar sensor system 54 comprises a plurality of radar sensor devices formed as radar transceivers (not shown). In this specific embodiment, the radar sensor system 54 is formed as a frequency-modulated continuous wave (FMCW) radar system configured to be operated at a radar carrier frequency that lies in a frequency regime between 20 GFIz and 90 GFIz, and more specifically in the frequency range between 57 GFIz and 64 GFIz, for example at a radar carrier frequency of 60 GFIz.

[0041] This specific embodiment of the disclosed sensor-based monitoring and/or surveillance system 50 is intended to be used in an automotive vehicle interior sensing system. The optical camera 26 and the radar transceivers of the radar sensor system 54 are directed towards a chest and an abdominal region of a vehicle driver for determining a breathing and/or a heartbeat activity.

[0042] The artificial neural network 12 is configured to provide, at the output side 16, an output signal 34 that represents a specific situation with regard to the vehicle driver to be monitored by applying at least one trained task to signals that have been received from the optical camera 26 and/or the radar sensor system 54, have been stored in the external memory unit 52 and that date back up to a predetermined time period. The predetermined time period has a length of up to 15 minutes, and in this specific embodiment amounts to about ten minutes.

[0043] The monitoring and/or surveillance system 50 is configured to monitor the vehicle driver’s health condition. The monitoring and/or surveillance system 50 operates based on the data received from the optical camera 26 and/or the radar sensor system 54 and, by applying at least one trained task to signals received from the optical camera 26 and/or the radar sensor system 54, concludes on the vehicle driver’s physical state in order to, for instance, but not limited to, recommend a break or to draw the driver’s attention to a potentially safety-relevant aspect. In order to assess the vehicle driver’s physical state, it is quite beneficial to have access to data of past driving situations. In case of monitoring the heartbeat activity with the radar sensor system 54, reference data for a regular heartbeat activity of the specific driver are used to infer whether or not a certain heartbeat is unusual for that person.

[0044] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments.

[0045] Other variations to be disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or“an” does not exclude a plurality, which is meant to express a quantity of at least two. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting scope.

List of Reference Symbols

10 sensor-based monitoring and/or surveillance system

12 artificial neural network

14 input side

16 output side

18 external memory unit

20 read head

22 write head

24 radar sensor system

26 optical camera

28 signal output line

30 Lidar device

32 signal output line

34 output signal

40 sensor-based monitoring and/or surveillance system

42 external memory unit

50 sensor-based monitoring and/or surveillance system

52 external memory unit

54 radar sensor system