Technical Field

The present invention relates to a system which enables to carry out real-time emotion analysis automatically during conversation in instant messaging applications and to report the emotional state (mood) of the party detected as a result of the emotion analysis carried out, to another party and/or parties included in the conversation.

Background of the Invention

Use of mobile devices and applications and services provided by means of mobile devices shows a continuous increase today. Instant messaging applications and services are one of the most popular applications and services being used by means of mobile devices. With use of instant messaging or instant messaging (IM) applications and services, users can send text, voice, video messages to each other and by establishing messaging groups created by more than two users if they want, they can message each other such that all users -who are group members- can receive and see messages. Today, parties who have conversation in instant messaging applications may need to use characters forming simple face shapes to reflect their emotions or face shapes (emojis) provided by the application. However, the other party cannot usually understand the actual emotional state of the person sending the message in this case. Therefore, there is need for a system which enables to carry out real-time emotion analysis automatically during conversation in instant messaging applications and to report the emotional state of the party detected as a result of the emotion analysis carried out, to another party and/or parties included in the conversation in order that the other party (message receiver) can perceive the real-time emotional state of the person sending the message, without difficulty in instant messaging applications in the state of the art.

The United States patent document no. US9402576, an application in the state of the art, discloses techniques comprising warnings and modifications which are used for electronic communication. In one embodiment of the said invention, an emotional state of a sender party/user is automatically detected; the emotional state data of the sender party is encoded; by adding the information of the inputs from at least one additional source to the encoded state data, the related data is changed and sent to the receiving party. Then, the metadata received from the sending party with the units being located in the receiving party is converted into a version that the receiver of the sender’s emotional state data can understand, by means of an extraction module. Transaction of converting the sender’s emotional state data is carried out by the emotion converter module disclosed in the invention.

Summary of the Invention

An objective of the present invention is to realize a system which enables to carry out real-time emotion analysis automatically during conversation in instant messaging applications and to report the emotional state of the party detected as a result of the emotion analysis carried out, to another party and/or parties included in the conversation.

Detailed Description of the Invention “A System for Carrying Out Real-Time Emotion Analysis in Instant Messaging Applications” realized to fulfil the objective of the present invention is shown in the figure attached, in which: Figure l is a schematic view of the inventive system.

The components illustrated in the figure are individually numbered, where the numbers refer to the following:

1. System

2. Camera

3. Mobile device

4. Instant messaging application

5. Instant messaging server

The inventive system (1) which enables to carry out real-time emotion analysis automatically during conversation in instant messaging applications and to report the emotional state of the party detected as a result of the emotion analysis carried out, to another party and/or parties included in the conversation; comprises:

at least one mobile device (3) which has at least one camera (2) configured to capture the user image and which is configured to run at least one application on it, establish connection with at least one remote server by using any remote communication protocol and to perform data exchange with the remote server over this connection established;

at least one instant messaging application (4) which is run on the mobile device (3); can establish connection with the camera (2) such that it can perform data exchange; and which is configured to enable users to message each other real-timely at least vocally and/or in writing, to trigger the camera (2) in order to take image about at least the face of the user during at least the messaging, and to receive the data about the taken images from the camera (2); at least one instant messaging server (5) which is configured to establish connection with at least one mobile device (3) by using any remote communication protocol, to perform data exchange with the instant messaging application (4) over this connection established with the mobile device (3), to ensure that data exchange is performed between the mobile device (3) of at least one user that the user sends message by means of the instant messaging application (4) during this data exchange, to detect real time emotional state of the user by receiving from the instant messaging application (4) the data about the image of the user -at least sending the message- received by means of the camera (2) and processing the said image data and the message data sent by the user, and to send the data about the detected emotional state to the instant messaging application (4) being run on the mobile device (3) of at least one other user in order to inform the at least one other user whom the user sent message by means of the instant messaging application (4) about the user’s emotional state detected.

The mobile device (3) included in the inventive system (1) comprises at least one camera (2) configured to take image about at least the face of the user and it is configured to run at least one instant messaging application (4) on it. The mobile device (3) is an electronic device such as mobile phone, smartphone configured to establish connection with at least one instant messaging server (5) by using any remote communication protocol in the state of the art and to ensure that data exchange is performed between the instant messaging application (4) and the instant messaging server (5) over this connection established. In one embodiment of the invention, the mobile device (3) is configured to establish connection with the instant messaging server (5) over a data network such as Internet and to ensure that data exchange is performed between the instant messaging application (4) and the instant messaging server (5) over this connection established. The instant messaging application (4) included in the inventive system (1) is run on the mobile device (3) and it is in connection with the camera (2) such that it can perform data exchange. The instant messaging application (4) is configured to perform data exchange in order to send and receive at least voice and/or text message with the instant messaging server (5) over the connection established between the mobile device (3) and the instant messaging server (5). The instant messaging application (4) included in the inventive system (1) is configured to establish connection with the camera (2) when the user starts making entry to the said instant messaging application (4) in order to send message, and to trigger the camera (2) in order to take image about at least the face of the user. The instant messaging application (4) is configured to receive from the camera (2) the data about the user image captured by the camera (2) after the message entry performed by the user is completed, and to send the data about the message entered by the user and the user image received from the camera (2) to the instant messaging server (5) over the connection established between the mobile device (3) and the instant messaging server (5).

The instant messaging server (5) included in the inventive system (1) is configured to establish connection with at least one mobile device (3) by using any remote communication protocol in the state of the art and to perform data exchange with the instant messaging application (4) over this connection established. In one embodiment of the invention, the instant messaging server (5) is configured to establish connection with at least one mobile device (3) over a data network such as Internet and to perform data exchange with the instant messaging application (4) over this connection established. The instant messaging server (5) is configured to ensure that message exchange is performed between the mobile device (3) of at least one user that at least the user sends message by means of the instant messaging application (4) during the data exchange performed by the instant messaging applications (4). The instant messaging server (5) is also configured to receive the user image data that are received from the instant messaging application (4) together with the message data and to detect the emotional state of the user by processing the message data and the user image data. The emotional state of the user may be real-time emotions of the user such as happy, sad, crying, laughing, afraid. The instant messaging server (5) is configured to send the information data about the detected emotional state after detecting the emotional state of the user, and the related message data helping to detect the emotional state, to the instant messaging application (4) that is run on the mobile device (3) of at least one other user to whom it is aimed to send message.

In one embodiment of the invention, the instant messaging application (4) is also configured to establish connection with the camera (2) when the user runs the instant messaging application (4) in order to read the incoming message, and to trigger the camera (2) in order that the image about at least the face of the user is taken. The instant messaging application (4) is configured to receive the data about the user image captured by the camera (2) after a predetermined period of time passes and to send the data about the message read by the used and the user image received from the camera (2), to the instant messaging server (5) over the connection established between the mobile device (3) and the instant messaging server (5). The instant messaging server (5) is configured to receive the user image data that are received from the instant messaging application (4) together with the message data and to detect the emotional state of the user by processing the message data and the user image data. The instant messaging server (5) is configured to send the information data about the detected emotional state after detecting the emotional state of the user, to the instant messaging application (4) that is run on the mobile device (3) of the other user sending the related message data helping to detect the emotional state.

In a preferred embodiment of the invention, the instant messaging server (5) is configured to keep the data about the user’s detected emotional state and the message and image data used for detecting this, under record. The instant messaging server (5) is configured to educate itself by processing the recorded data at predetermined periods and to improve the detection of user’s emotional state that it will perform hereafter. In a preferred embodiment of the invention, the instant messaging server (5) is configured to process the recorded data by using deep learning techniques and to improve the detection of user’s emotional state that it will perform hereafter.

In one embodiment of the invention, the instant messaging server (5) is configured to keep the data about the user’s detected emotional state under record together with a distinctive token and date and time tag about the user. In one derivative of this embodiment of the invention, the instant messaging server (5) is configured to present the data about the detected emotional states of the recorded users, to the users historically over the related instant messaging applications (4).

With the inventive system (1), emotional states of users who have conversation with each other vocally and/or in writing over the instant messaging applications (4) both while they sent the message and while they read the incoming message are automatically detected depending on the facial expression of the user at that moment or the content of the message sent/read and the users are automatically informed about the real-time emotional states detected. Thereby, it is not necessary for a user to use faces created by characters that are not entirely comprehensible or template face shapes provided by the instant messaging applications (4) in order to indicate his/her emotional state and this enables the other party not to be confused about the user’s emotional state. Due to the fact that the instant messaging server (5) educates itself particularly by deep learning techniques in the inventive system (1), the said instant messaging server (5) can perform the emotional state detection more precisely and more accurately in time.

Within these basic concepts; it is possible to develop various embodiments of the inventive system (1); the invention cannot be limited to examples disclosed herein and it is essentially according to claims.