The present invention relates to transmission of media and in particular to the organization of data transmitted in a media stream. Background of the invention

Video distribution systems conventionally include a video source and at least one receiving device to receive video content. The video content may be distributed over a network, such as broadcast television, Over The Top (OTT) delivery, Internet Protocol Television (IPTV), etc. Network-based media delivery services typically code an input video sequence as coded video data that has been parsed into a plurality of separately deliverable segments.

HTTP Live Streaming (HLS) is the most widely adopted Internet video delivery protocol and it is widely supported in mobile operating systems and modern browsers. This makes it currently a de-facto standard for OTT video delivery. Instead of streaming the video from a server to a playback client, the client is essentially downloading video as multiple HLS segments using HTTP protocol. The length of a HLS segment is typically equivalent to a playback time of 10 seconds. The relatively large file length, together with the fact that the client device may often select from multiple bitrates, makes the HLS resistant to short-term congestion. This makes HLS a very favorable technique to stream video files stored in a network server.

HLS can also be used for live streams, but it has a considerable latency because the built-in playback components of a video player typically buffer 2 - 3 HLS segments before the video playback is started. Considering all delay components of the system, the end-to- end latency typically increases to 30 - 40 seconds. For video systems requiring nearly real-time playback, such as various surveillance systems, such delay is too long. Summary of the invention

Now there has been invented an improved method and technical equipment implementing the method, by which the above drawbacks can be at least significantly alleviated. Various aspects of the invention include methods, an apparatus, computer programs and a system, which are characterized by what is stated in the independent claims. Various embodiments of the invention are disclosed in the dependent claims. According to a first aspect, a method according to the invention is based on the idea of obtaining multimedia data arranged in segments according to a streaming protocol; dividing the multimedia data into a plurality of data chunks; creating metadata relating to content of each data chunk; and interleaving the metadata and the data chunks into a transmission frame having structure such that the metadata of the associated data chunk precedes said associated data chunk.

According to an embodiment, the multimedia data is arranged in segments according to HTTP Live Streaming (HLS) protocol or Dynamic Adaptive Streaming over HTTP (DASH) protocol.

According to an embodiment, the method further comprises generating a plurality of data chunks per one second of playback time. According to an embodiment, the metadata comprises a security record associated with the data chunk and content metadata of the data chunk.

According to an embodiment, an encryption key is associated with security record.

According to an embodiment, the encryption key is obtained from a trusted third party. According to an embodiment, the data chunks are encrypted using a streaming protocol specific encryption scheme. According to an embodiment, the security record comprises one or more of the following:

- a field for identifying the type of the security record;

- a field for a network address, such as an URL of the trusted third party;

- a field for an initialization vector for decryption;

- a field for identifying the content associated with the security record.

According to an embodiment, the content metadata comprises one or more of the following:

- a field for identifying the content associated with the content metadata;

- a field for identifying the type of content;

- a field for additional data, such as playback duration and start time of the content

According to a second aspect, there is provided an apparatus comprising: a processor; and a memory unit operatively connected to the processor wherein the apparatus is configured to: obtain multimedia data arranged in segments according to a streaming protocol; divide the multimedia data into a plurality of data chunks; create metadata relating to content of each data chunk; and interleave the metadata and the data chunks into a transmission frame having structure such that the metadata of the associated data chunk precedes said associated data chunk.

According to a third aspect, there is provided a computer program product, embodied on a computer-readable medium, for arranging multimedia data for distribution, comprising computer code adapted to perform: obtaining multimedia data arranged in segments according to a streaming protocol; dividing the multimedia data into a plurality of data chunks; creating metadata relating to content of each data chunk; and interleaving the metadata and the data chunks into a transmission frame having structure such that the metadata of the associated data chunk precedes said associated data chunk. These and other aspects of the invention and the embodiments related thereto will become apparent in view of the detailed disclosure of the embodiments further below.

List of drawings

In the following, various embodiments of the invention will be described in more detail with reference to the appended drawings, in which

Fig. 1 shows a functional block diagram of an exemplary streaming system;

Fig. 2 shows how the media streams, such as video streams, are organized into HLS segments;

Fig. 3 shows a flow chart of a streaming method according to an embodiment of the invention; shows a principle of dividing HLS segments into data chunks according to an embodiment of the invention; illustrates obtaining encryption keys for the data chunks encrypted with the security record according to an embodiment of the invention;

Figs. 6a, 6b show examples how the data chunks and the associated metadata may be multiplexed into the frame structure according to an embodiment of the invention; and

Fig. 7 shows an example streaming system according to an embodiment. Description of embodiments

In the following, the invention will be illustrated by referring to HLS streaming systems. It is, however, noted that the invention is not limited to HLS solely, but it can be implemented in any similar system as the technology advances. For example, Dynamic Adaptive Streaming over HTTP (DASH), (a.k.a. MPEG-DASH) streaming system may be utilised in the implementation.

Figure 1 shows a functional block diagram of an exemplary streaming system 100, wherein the embodiments described below may be implemented. The system 100 may include a media server 1 10, a distribution server 120 and a communication network 130 for delivering media data to one or more client devices 140, 142, 144. From the perspective of the client devices, the media server 1 10 and the distribution server 120 may be considered together to form a media source 150.

The media server 1 10 takes input streams of media and the encoder 1 12 encodes the streams according to a predetermined encoding scheme and encapsulates the encoded streams in a format suitable for delivery. In current HLS systems, the streams are encoded according to MPEG-2 encoding and they are encapsulated as MPEG-2 Transport Streams (MPEG-2 TS). The segmenter 1 14 organizes the encoded input media stream as a plurality of coded segments 1 16a -1 16n and generates a playlist 1 17 to identify the segments. In HLS systems, the coded segments may also be referred to as HLS segments. The input media stream may also be readily in the MPEG-2 TS format, in which case the encoded input media stream may be provided directly to the segmenter 1 14.

Each coded segment may be stored by the media server 1 10 at locations that can be referenced by unique URLs (Uniform Resource Locators). Each coded segment is indexed by the media server, whereupon e.g. the URL, a program date and time are associated with the coded segment in an index file. The playlist is created on the basis of the index files, and it identifies the available media streams and the locations of each segment as stored at the media source. The playlist 1 17 and coded segments of the available media resources 1 16a -1 16n are stored in data storage 1 18, which is accessible by the distribution server 120. The playlist is published to the client devices and it includes a plurality of tags that provide information to the client regarding the content of the media stream.

The distribution servers 120 are typically standard web servers. It is noted that while Figure 1 shows the data storage 1 18 as residing in the media server, it may as well be provided in the distribution server or in both. The distribution server 120 is responsible for accepting client requests and delivering prepared media and associated resources to the client. The distribution server 120 maintains and publishes URLs of the playlists of the media source 1 10, on the basis of which the client requests may be generated. Upon a request from a client device 140, 142, 144, the distribution server 120 may transmit the coded segments 1 16a - 1 16n via one or more channels of the communication network 130.

For streaming applications involving live video, the media server 110 and/or the distribution server 120 may buffer coded video segments for a predetermined amount of time. The buffer is continually updated such that older segments are deleted from the buffer as new segments are generated and stored. Similarly, the playlist 1 17 is updated over time according to sliding window principle to reflect the updated content of the buffer.

The client devices 140, 142, 144 may be provided with standard media players that request and download coded segments from the distribution server 120. The coded segments are decoded and rendered for playback. The client device downloads the playlist 1 17 from the distribution server 120, whereupon the client device may request a particular coded segment for delivery and decoding. After decoding and rendering the requested coded segment, the client device may request the next entry in the playlist 1 17 from the distribution server 120. The client device may periodically refresh its copy of the playlist 1 17 and continue with the above process by downloading further coded segments from the distribution server 120 until the client device discontinues playing the media stream.

It is noted that the streaming system 100 of Figure 1 is only shown for illustrative purposes, and the actual configuration, topology and architecture of the streaming system may vary in multiple ways. For example, a distribution server 120 may transmit multiple (different) media streams to multiple client devices. On the other hand, the distribution server 120 may transmit a common media stream to multiple client devices using different bit rates and/or different frame sizes according to playback capabilities of different types of client devices. Moreover, various network architectures may be used between the media source 150 and the client devices 140, 142, 144. For example, there may be gateway devices or routers providing a connection to the client device by a wired or wireless local area network.

Figure 2 illustrates how the media streams, such as video streams, are organized into HLS segments. A video stream may be represented as a sequence of individual source frames which may be organized into video segments. The sequence of individual source frames may be encoded according to one or more compression algorithms in order to obtain a sequence of compressed frames with a reduced data rate as compared to the source frames. In the HLS, the encoded video frames are encapsulated into MPEG-2 TS packets, and a plurality of MPEG-2 TS packets (p1 , p2, pn) are combined together to form a HLS segment.

The length of a HLS segment is typically equivalent to a playback time of 10 seconds. The relatively large file length, together with the fact that the client device may often select from multiple bitrates, makes the HLS resistant to short-term congestion, and therefore, HLS can easily adapt to changing delivery conditions of unmanaged Internet. This makes HLS a very favorable technique to stream video files stored in a network server.

HLS can also be used for live streams, but it has a considerable latency because the built-in playback components of the operating system or the browser typically buffer at least two, usually three HLS segments before the video playback is started. Considering all delay components of the system, the end-to-end latency typically increases to 30 - 40 seconds with the default 10 second HLS segment size.

For HTTP streaming the latency may be reduced by decreasing the HLS segment size to 1 - 2 seconds. However, this has the adverse effect on increasing the number of video segments to be stored in data storage and indexed in a database. Also the number of HTTP operations increase, which will increase the cost of using cloud platforms. Lastly, the latency is still in the region of 5-10 seconds and the playback operation may be susceptible to an additional latency, because the playback component behaviour is optimised for the nominal 10 second segment size.

Now, an improved HLS streaming method is introduced herein for enabling a significant reduction in the end-to-end latency when playing live material, while still maintaining full HLS compatibility for storage and playback of recorded material.

In the method, which is shown in Figure 3, multimedia data arranged in segments according to a streaming protocol, such as in HTTP Live Streaming (HLS) segments, is obtained (300). The multimedia data is divided (302) into a plurality of data chunks (a.k.a. sub-segments), and metadata relating to content of each data chunk is created (304). The metadata and the data chunks are interleaved (306) into a transmission frame structure such that the metadata of the associated data chunk precedes said associated data chunk.

Thus, a transmission frame structure is provided which enables the playback components of a HLS player to start the decryption and the playback already when a fraction of a HLS segment has been received. The sender of the multimedia data arranges the multimedia data into said frame structure, whereupon the receiver may start the decryption and the playback of the data chunks as soon as a predetermined number of data chunks and their associated metadata have been received. The decryption may be started when at least one encrypted block is received, but practically the decryption may be carried out for a whole data chunk. The playback may depend on the used player, for example for HTML5 players, a Group- Of- Pictures (GOP) may be needed before starting the playback.

Accordingly, even though the HLS playback components operate, for example download and buffer, with complete HLS segments, the decoding and playback may start even when only a few data chunks have been received. Hence, the end-to-end latency in the playback of live streams may be significantly reduced, for example into 2 - 3 seconds in total.

Figure 4 illustrates the principle of dividing the content of the HLS segment, organized as shown in Figure 2, into data chunks. It is noted that the lengths of the data chunks in Figure 4 are not in scale with the length of the HLS segment. According to an embodiment, a plurality of data chunks is generated per one second of playback time. At minimum, the data chunk may consist of only one video frame, which corresponds to a playback time of 1/25 second (40 ms). The length of the data chunks may be constant or it may be dynamically adjustable within a HLS segment.

According to an embodiment, the metadata comprises a security record associated with the data chunk and content metadata of the data chunk. When the HLS segment is divided into data chunks, each data chunk may be encrypted with an encryption key. The encrypted data is associated with a security record that is used to derive the encryption key and determine the authorization for the decryption of the encrypted data chunk.

Figure 5 illustrates the principle of obtaining the encryption key when the playback component of a player 500 receives the data chunk 502 encrypted with the security record 504. The encryption keys may be managed by a trusted third party 506, from which the playback component may query the encryption key for the particular data chunk. The playback component, or a decryption application 508 therein, sends 510 its identification and the security record 504 received in the metadata to the third party. An application 512 of the trusted third party authenticates the playback component and derives the encryption key based on the identification and the information of the security record. The trusted third party sends 510 the encryption key to the playback component, which uses the encryption key to obtain decrypted content 512 of the data chunk.

According to an embodiment, the encryption key is obtained from the trusted third party using a HTTP GET operation.

According to an embodiment, the data chunks are encrypted using Advanced Encryption Standard AES-128 encryption scheme.

According to an embodiment, the security record comprises one or more of the following:

- a field for identifying the type of the security record;

- a field for a network address (e.g. an URL) of the trusted third party e.g. for performing the GET operation;

- a field for an initialization vector for AES-128 decryption;

- a field for identifying the content associated with the security record.

According to an embodiment, the content metadata comprises one or more of the following:

- a field for identifying the content associated with the content metadata;

- a field for identifying the type of content;

- a field for additional data, such as playback duration and start time of the content.

Figures 6a and 6b show two examples how the data chunks and the associated metadata may be multiplexed into the frame structure. In Figure 6a, the metadata N associated with the data chunk N precedes the data chunk N. Similarly, the metadata N+1 associated with the data chunk N+1 may precede the data chunk N+1 , etc. In any case, the associated metadata is preferably received before the encrypted data chunk such that the encryption key may be derived from the trusted third party in advance. Because of the transaction latency in communicating with the trusted third party, at least the security record part of the metadata may be provided earlier in the frame structure, for example several seconds before the encrypted data chunk. Thus, the player may obtain the encryption key in advance and start the decryption as soon as it receives the data chunk corresponding to the encryption key.

In Figure 6b, both the metadata N associated with the data chunk N and the metadata N+1 associated with the data chunk N+1 precedes the data chunk N. Similarly, the metadata N+1 associated with the data chunk N+1 and the metadata N+2 associated with the data chunk N+2 may precede the data chunk N+1 , etc. This multiplexing scheme may be used to ensure that the encryption keys of the data chunks are received in time. The encryption key may be guaranteed to be unchanged such that no new transactions with the trusted third party are needed.

The sender may preferably transmit the data chunks and the associated metadata as soon as they become available. The receiver of the transmission frame may be e.g. a player having necessary playback components or an intermediate server performing forwarding and/or storing of the data. If the receiver of the data is a player, the player may perform a method comprising: receiving a transmission frame comprising multimedia data divided into a plurality of data chunks and metadata of each associated data chunk preceding said associated data chunk; buffering the multimedia data of a predetermined amount of data chunks de- multiplexed on the basis of said metadata; and re-multiplexing the multimedia data into data fragments suitable for streaming playback. According to an embodiment, the multimedia data is re-multiplexed into MPEG4 fragments.

According to an embodiment, the data chunks are encrypted using a streaming protocol specific encryption scheme and the metadata comprises an association to an encryption key, wherein the demultiplexing comprises obtaining the encryption key from a trusted third party, and decrypting the encoded data chunks for buffering. Thus, the data chunks are decrypted, buffered, decoded and displayed as they become available. The playback may start as soon as a predetermined amount of decrypted data chunks is buffered, e.g. corresponding to a playback time of 2 - 3 seconds. For web browser playback, HTML5 Media Source Extensions may be used to re- multiplex the data from the data chunk into a format (such as MPEG4 fragments), which can be played by the HTML5 video player. Here the typical buffering size is one Group-Of-Pictures (GOP), which may correspond to duration of one second. If the receiver of the data is a server performing store-and-forward, the data chunk may be forwarded to the next component, such as another intermediate server, a gateway or a player, immediately after the data chunk is received. For storage, the data chunks are buffered until data of a complete HLS segment is received. The data is re-assembled into a regular HLS segment, and the HLS segment is stored in a data storage and indexed in the database. As a result, the stored data is in the exact format needed for efficient HLS video delivery.

The framing principle supports multiple methods of transport including HTTP/HTTPS and RTP/SRTP ((Secure) Real-time Transport Protocol). The selected transport method may depend on the underlying communication system: for example, RTP can be used in point-to-point and point-to-multipoint communication over Local Area Network or Internet, whereas HTTP can be used in client-server architectures over Internet. Figure 7 shows an example streaming system according to an embodiment, where a sender 700 creates streams, which are divided into data chunks as described above, and sends them to a store-and- forward server 702. There are two players with different capabilities arranged to receive the streams: player 704, which is capable of processing the chunked streams almost in real time, herein referred to as HLS-RT (Real Time) player, and a conventional HLS Player 706, which capable of only play back regular (non-chunked) HLS streams. The operation of the store-and-forward server 702 depends on the capabilities of the player. The store-and-forward server 702 relays the chunked stream as such to the HLS-RT player 704. For the conventional HLS Player 706, the server 702 reassembles the HLS segments from the data chunks for storage and forwards the HLS segments to the HLS Player 706.

The system shown in Figure 7 may be utilised in various video surveillance systems, for example. The sender 700 may be provided with a surveillance camera capturing video data about an area to be monitored. The video data may be encoded into a format suitable for streaming, such as in H.264 format. The sender 700 produces a stream divided into data chunks as described above, and sends it to the store-and-forward server 702. Herein, WebSockets may be used in the transportation, where the data chunks are transferred in binary frames and the metadata in text-based frames.

The transportation from the server 702 to the HLS-RT player 704 may use WebSockets to implement HTTP compatible push streaming. The framing on top of the WebSockets may be implemented with a suitable lightweight protocol, such as STOMP (Simple Text Oriented Message Protocol). The server 702 relays the data chunks and the metadata to HLS-RT player 704 immediately upon reception. HLS-RT Player 704 may use the Media Source Extension to re-multiplex the received chunks into MPEG4 media fragments. HTML5 video element may be used for playback of the fragments. The transportation from the server 702 to the conventional HLS player 706 is implemented as normal HLS, where the server 702 provides playlists and media segments for the player to download. In such implementation, the latency from the sender (camera) 700 to the HLS-RT player 704 is typically about two seconds, whereas the latency from the sender 700 to the conventional HLS Player 706 is typically about 40 seconds. The system of Figure 7, due to being rather straightforward and inexpensive to implement may be utilized, for example, in various home surveillance systems. The camera may provide video data, for example, about home indoor/outdoor area, as a door phone video or surveillance data in elderly peoples' home care system. The end-to- end latency in the chunked data stream is so minimal that the video data can be followed practically in real-time. On the other hand, the store-and-forward server stores the streams as re-assembled to regular HLS segment, whereupon the surveillance data may be checked afterwards to address any incidents.

A skilled person appreciates that any of the embodiments described above may be implemented as a combination with one or more of the other embodiments, unless there is explicitly or implicitly stated that certain embodiments are only alternatives to each other.

It is noted that the Figures referred above illustrate functional block diagrams of exemplary systems according to embodiments of the invention. In implementation, the systems may be embodied as hardware, in which case, the illustrated blocks may correspond to circuit sub-systems within the systems. Alternatively, the components of the systems may be embodied as software, in which case, the blocks illustrated may correspond to program modules within software programs. In yet another embodiment, the systems may be hybrid systems involving both hardware circuits and software programs.

Some embodiments may be implemented, using a non-transitory computer-readable storage medium or article which may store an instruction or a set of instructions that, if executed by a processor, may cause the processor to perform a method in accordance with the disclosed embodiments. The exemplary methods and computer program instructions may be embodied on a non-transitory machine- readable storage medium. In addition, a server or database server may include machine-readable media configured to store machine executable program instructions. The features of the embodiments of the present invention may be implemented in hardware, software, firmware, or a combination thereof and utilized in systems, subsystems, components or subcomponents thereof. The machine- readable storage media may include any medium that can store information. Examples of a machine-readable storage medium include electronic circuits, semiconductor memory device, ROM, flash memory, erasable ROM (EROM), floppy diskette, CD-ROM, optical disk, hard disk, fiber optic medium, or any electromagnetic or optical storage device.

It is obvious that the present invention is not limited solely to the above- presented embodiments, but it can be modified within the scope of the appended claims.