The MultiMedia System

The MultiMedia system stores and manipulates patient data in the Oncology department of the Soroka hospital. The database contains various types of multi-media information including textual, images and voice.

The Structure of the MultiMedia System.

• Interface layer provides grapical user interface for viewing data, querying the database and running some of the MultiMedia algorithms. It is implemented in Java.
• Intermediate layer connects Interface layer with other layers: it runs some of the MultiMedia algorithms and connects to the Database server. It receives messages from the Interface layer, performes corresponding actions and sends results to the Interface layer. The actions are:
  • performing a SQL query (received from the interface layer) for retrieving data from the database or altering the database (inserting, updating of rows),
  • running MultiMedia algorithms.
  The Intermediate layer is implemented in C with Embedded SQL.
• MultiMedia algorithms layer consists of the following algorithms:
  • Image processing:

    Compression of images (is run by the Interface Layer),

    Tumor identification (is run by the Interface Layer),

    Texture-based Image Segmentation (is run by Interface and Intermediate Layers).

  • Voice processing:

    Speech Keyword processing (is run by Intermediate Layer).

  • General MultiMedia query:

    Query on all data types: text, image and voice.

  Description of the algorithms is presented in the next section.
• Database server layer: - Informix relational DBMS. Receives queries from the intermediate module, performs them and returns results to the intermediate module.
• Database layer: the MultiMedia database consists of four patients of the oncological department.

MultiMedia Algorithms.

Speech Keyword processing.

1. Adding new keyword to the database includes the following steps:
   • Recording the new keyword, neighbors of the keyword (words that are similar to the keyword) and "garbage" words (words that are not similar to the keyword). Each word is recorded many times.
   • Training the new keyword and its neighbors. This process generates a model of each trained word.
   • Joining the models of the keyword and its neighbors to generate the join model of the keyword.
   • Finding a threshold of the probability that the keyword fits the model, using garbage words.
   • Finding a size of a check window for recognition, by calculating the average length of keywords.
   • Finding the distance from the beginning of one check window to the next one.
   • Saving in the database a new keyword model, its threshold, its size of a check window, and its distance between check windows.
2. Recognition of a keyword in a speech file is performed in the following way:

   For each check window in the speech file competition between all models in the join model of the keyword is made. The probability of fitness of each model to the check window is calculated. If some keyword model has the greatest propability then it is compared with the threshold. (that propability can be stored in the database for future reference.) If the probability is greater that the threshold than the check window contains the keyword.

Image processing and Tumor identification.

1. Image Processing allows the following actions:
   • Zooming / Unzooming.
   • Coloring the black and white image. This provides a better view of some spots on the image that were not seen before with an unarmed eye.
   • Scaling the image and getting sizes in millimeters of the inside image objects.
   • Automatically detecting the tumor, using region growing algorithm. A user specifies a start rectangle inside a tumor in an image. The algorithm calculates the average color of the pixels in the rectangle and then expands the region to the maximum size of the tumor, referring to the alpha parameter specified by the user. Each pixel in the border of the rectangle is examined. If its color is in the range that is specified by the alpha parameter this pixel is added to the rectangle. This action recursively continues until there are no more pixels to add. The gathered pixels form the tumor.
   • Manually editing the tumor region, by adding or deleting pixels.
   • Comparing tumors parameters and details.
   • Coloring the tumors.
   • Saving in database tumors parameters for each image.
2. Image Comparisons are performed in the following way:

   A user selects a tumor in an image and the system automatically finds the most similar tumors in a set of images. The finding algorithm gets parameters of the selected tumor and searches for tumors with the same parameters (up to an epsilon, specified by the user).The tumors are determined by the algorithm automatically, without specifying the start rectangle. The images, where the similar tumors were found, are ordered with the most fit image first. The user can load each image and view the found tumors.

Texture-based Image Segmentation.

1. Perform a segmentation on an image.
   The result of a segmentation is a list of regions, where a region is characterized by a contour and a parameterized representation of the texture within the contour. Parameters of segments are stored in the database.
   Two kinds of segmentation are provided by the project:
   • Automatic segmentation.
     It divides an image according to average colors in different regions: each segment is composed of many small regions with close average colors. The algorithm is based on two well-known algorithms: "Split and Merge" and "Region Growing". It consists of three stages:
     • partition into homogeneous regions
     • merging adjacent regions that have similar average colors
     • removing small regions
   • Manual segmentation.
     There is a graphical interface that allows a user to define segments manually. Manual segmentation may be also used as a way to change results of automatic segmentation.
2. Given a segmentation of an image it is possible to perform the following actions:
   • edit the contours in a topological preserving manner
   • mark some counters as non-textures
   • view parameters of a segment
3. Image reconstruction: from a given indexed representation create an image, including the combination of texture and non-texture regions.

   Reconstruction is based on rectangles that bind different segments. Each rectangular is presented by a matrix where each entry contains a color of a pixel, if the pixel is inside the segment, or a negative value, if the pixel is outside the segment. These matrixes are merged into one large matrix of the image in a way that only positive values (with colors of pixels) are copied from small matrixes to the large matrix. The large matrix presents the reconstructed image.

4. Presentation of parameters of all segments in an image.

Image Compression.

General MultiMedia query

There are three types of data: text, image and voice. Possible queries for each of these types are:

• for text data: select all patients that satisfy some constraints
• for image data: select all patients that have an image with a tumor similar to the chosen tumor.
• for voice data: select all patients that have a voice file with the chosen keyword.

Results of each query are graded sets. Each patient is assigned a grade of satisfaction to the query. A grade is a real number in the interval [0,1], which identifies the fitness of a patient to the query. All patients that satisfy a text query are assigned grades equal to 1. The results of the image and voice queries are lists of patients, sorted by their grade, where the larger the grade is, the better the match is.

A grade of a patient that satisfies image (voice) query, is taken to be the maximum grade of all images (voice files) of this patient, that contain a similar tumor (a keyword). The reason is that to characterize a patient the best image (voice file) is taken.

Now let's consider a general query which is the conjunction of these kinds of queries. The result of the conjunction contains the patients that are in the results of all these queries. Each such patient gets a grade, which is the minimum of the grades of this patient in all queries of the conjunction. Therefore, the result of the general query is the ordered list of patients, sorted by the resulted grades of these patients.