2、Facial Image Recognition [1]PaPeRo uses two CCD cameras located at the actual position of the eye to capture images, analyze the images captured to find the position of a face, and identify the person’s face by matching the facial image with the ones previously registered to find the same person.

3、Facial Image Recognition
PaPeRo uses the following methods to recognize facial images:



Facial Image DetectionFirst, shapes that look like eyes are searched for in the images captured by the PaPeRo CCD cameras. Then, decisions are made regarding “likeness of the face” for areas that include the candidate eye areas found. If a certain level of likeness is obtained for the image, it is recognized as a face.
If two or more objects in an image are determined to be face-like objects, it may be recognized that there are actually two or more faces.
In addition, PaPeRo can determine whether the person is nearby or faraway by measuring the distance to his/her face using images captured by the two CCD cameras (stereoscopy*).

Facial Image MatchingAfter a facial image is detected, face matching is performed to identify the person who has the face.
Facial image matching is performed by comparing the facial area detected with about 30 facial image entries (templates) registered in a database.
In this database, facial images taken not only in a bright room but also in a dark room can be included to ensure accurate recognition of facial images under different illuminated conditions of the room.

• * “NeoFace,” a product of NEC, is used for facial image detection and matching functions.

Stereo Vision
With stereo vision it uses triangulation from the images recorded with the two CCD cameras and obtains the distance to the objects projected.

4、PaPeRo can recognize handwritten characters and printed characters on a piece of paper using optical character reader (OCR) functions.

Using this function, you can make PaPeRo remember names and words by showing a piece of paper with characters on it instead of using an entry pad. You can also show PaPeRo a mathematical formula or a quiz problem written on a piece of paper to make PaPeRo calculate the answer or reply; more natural and enjoyable communications may be enabled. * Character recognition technologies of NEC are used at post offices in mail handling machines that automatically read addresses on envelopes and postcards to sort them by delivery destinations.

6、Gesture DetectionPaPeRo is implemented with a gesture detection function to detect a person waving his/her hand and to move towards the direction of waving. First, any moving object is found and then size and skin color information is checked to see if an actual hand is being waved or not.
PaPeRo can detect a person waving his/her hand from a distance of 4 to 5 meters.

Pendulum DetectionPaPeRo can also find a swinging pendulum. First, PaPeRo detects the amplitude of the pendulum motion. When the detection is complete, it begins swinging the neck synchronizing with the pendulum. Then, while swinging the neck and following the motion of the pendulum, PaPeRo falls to sleep, like in a hypnotic trance.
This function resulted from the idea that it is desirable to turn PaPeRo off in a more natural way, rather than to press a button to mechanically stop the operation.

Obstacle DetectionPaPeRo can detect an obstacle that is within 3 meters using the 2 CCD cameras. Also, PaPeRo can recognize obstacles in various environments by combining stereoscopy (for measuring the distance to obstacles) and patterns of walls and floors.

• * Regarding stereoscopy, refer to item (3) of “Image Recognition Technology.”

二、Speech Recognition Technolog / Speech Recognition

1、Speech Recognition TechnologySpeech recognition technology is used to distinguish and identify what is said by a person. By using the speech recognition technology, PaPeRo can correctly recognize an instruction such as “Please dance” to start dancing as illustrated below.

PaPeRo is equipped with the microphones and loudspeakers shown in the figure below. Any language spoken into these microphones is checked against the vocabulary entries in a speech recognition dictionary to select the closest speech expression. Note that any word or sentence may be registered in this speech recognition dictionary provided that it is in Japanese.

Here, the mechanism of speech recognition is described in more detail.

2、Speech Recognition FlowHere, speech recognition processes are briefly described. The figure below shows the flow of speech recognition processes. As shown, the software program that performs speech recognition processes is called a speech recognition engine.
In this example, entries registered in the speech recognition dictionary include the five sentences: “How are you doing?” “Please dance,” “Good morning,” “Good afternoon,” and “Right.” When a person says “Good afternoon” into the microphone, the corresponding voice data is recorded in the computer in the form of digital data.
Only actually pronounced portions in the recorded voice data will be extracted, discarding any unnecessary parts. This task is called the voice segment detection process. The output voice data from the voice segment detection process will be checked against the templates generated from the speech entries in the speech recognition dictionary and the nearest one will be selected as the speech recognition result.
Note that some speech recognition software does not provide recognition results if it is decided that no speech entry in the dictionary matches the input speech. This function is called the “rejection function.” If no speech recognition results are determined, it will be “rejected.”

While the above figure suggests a situation where voice data is compared directly, it should be noted that the input voice data will actually be divided into a number of small segments, as shown in the figure below, to check the sequential arrangement of individual features of the segments.

3、Features of PaPeRo Speech RecognitionThe performance objective of PaPeRo speech recognition is not to cause any stress to the users when the system is used in a real home environment. A typical user environment is assumed to be a living room. In some cases, the user and PaPeRo could be facing each other to exchange dialogues. In other cases, the user could be sitting on a couch talking with PaPeRo. Also, in an actual home, it will typically be the case that more than one person are living together generating various kinds of sounds such as footsteps, conversation, household noise, television sound, and so on.
The speech recognition functions of PaPeRo have the following features in order to realize successful speech recognition in such a home environment.

Support for unspecified speakers		Hands free interface		Distance free interface
The robot can engage in conversation without specifying a user.		No special microphones or headphones are used.		There is flexibility in the distance and a certain amount of recognition is possible even when utterances are made from a little far away.

As such, the primary features of the speech recognition function of PaPeRo are hands free operation and adaptability to various speech distances. However, in adapting to various speech distances, it is unavoidable for PaPeRo to hear unnecessary sounds (e.g., conversation or noise) that are generated in the distance. Such unnecessary sounds will have a harmful effect on speech recognition.
For this reason, PaPeRo is equipped with the following functions to distinguish a specific sound source from multiple sound sources.

• •Echo canceller
• •Noise canceller
• •Rejection dictionary

These functions will be briefly introduced next.

4、Echo CancellerFormer versions of PaPeRo didn’t have the capability to recognize speech while PaPeRo itself was speaking. While PaPeRo was speaking, the voice generated from the loudspeaker of PaPeRo was picked up by the microphone together with the speech of the target person and it was difficult to distinguish the target voice. So, it was necessary to talk to PaPeRo only while the LEDs on the ears were lit (when PaPeRo is not generating any sound and speech recognition is functional).
To solve this problem, echo canceller technology was applied to enable speech recognition even while PaPeRo is speaking. This echo canceller function removes PaPeRo’s voice components that are estimated to exist in the microphone signals of PaPeRo, using the signals sent to the loudspeaker when PaPeRo speaks.
Using this function, now PaPeRo can extract only the targeted voice signals even when its own speech is overlapping with someone else’s.

5、Noise CancellerSpeech recognition in a noisy environment (e.g., loud television) was very difficult with the former versions of PaPeRo although speech recognition was successful in a quiet room. This is because of the difficulty to distinguish only a specific sound source (i.e., the target human voice) from existing multiple sound sources or noise, other than a human voice.
Here, speech recognition performance in a noisy environment is dramatically improved and stabilized through the use of a new technology, the noise canceller technology. This noise canceller function removes estimated noise components from the voice microphone signal by using multiple microphones in addition to the voice microphone.
With PaPeRo, a noise microphone is installed on its back in addition to the voice microphone on its head. The noise canceller removes any sound components coming from the sides and/or back of PaPeRo, which are also estimated using the sound input to the rear microphone.
Using this function, now PaPeRo can make out only the voice signals coming from the front of PaPeRo even when noise is present in the surrounding area.

6、Rejection DictionaryThe rejection dictionary is a speech recognition dictionary used to provide recognition results when it is decided that no word entry is close to the input word. The rejection dictionary enables one of the functions to reject unnecessary sounds, and is primarily useful for preventing erroneous responses to noise signals. The rejection dictionary is designed so that an accurate input of speech will be properly recognized and should not be recognized for rejection and that any other unnecessary sounds are exhaustively recognized for rejection, to every extent possible.
While the rejection function itself is already implemented as one of the functions of the speech recognition engine used, the performance of the rejection function alone may be insufficient for ensuring the hands free and distance free environment targeted by PaPeRo. Accordingly, it is now designed to satisfy both speech recognition and rejection accuracies at the same time by using a rejection dictionary that is specialized for the speech recognition vocabulary entries to deal with unnecessary voice signals.