learnCustomEntities()

learnCustomEntites( patterns, config ) → {count of patterns}

This method is used for defining the list of custom entities to be extracted as a pattern match from a winkNLP doc. It returns a count of patterns learned as defined in the first argument patterns. For example,

const patterns = [
  { name: 'nounPhrase', patterns: [ '[|DET] [|ADJ] [NOUN|PROPN]' ] }
];
const count = nlp.learnCustomEntities( patterns )
console.log( count );
// -> 1

The learnCustomEntities() method must be called before readDoc() to ensure custom entities’ detection.

patterns

Pattern consists of any combination of literals and token annotations. The annotations are either system entities or POS tags. The annotated value is always in UPPERCASE.

A pattern can contain a number of useful token properties such as POS tags NOUN, VERB, ADJ, URL etc. For example using POS tag ADJ in a pattern will match any of the words such as blue, green, large, medium or fast.

A Shorthand Pattern allows definition of a custom entity in a single text string. For example, define a location as new [york|delhi|orleans|brunswick].

Shorthand Patterns can also include token’s annotations — POS tag and Entity. The resulting entity will contain one or more grouped tokens.

In a pattern, a single token or part-of-speech tag matches with a single token, whereas an entity may match with one or more tokens.

• 1 large fries or One small fries is matched using the pattern sequence CARDINALEntity, ADJPOS tag, friesLiteral.
• Pickup at 5:30PM is matched using PickupLiteral, atLiteral, TIMEEntity.

During the learning, one can also use the mark( begin index, end index ) method to extract the entity from a matched pattern. Let’s take the example text: “My beautiful friend lives in a small cottage with her fluffy cats and playful dogs.” To extract all adjective-noun pairs, the pattern will be defined as follows:

const patterns = [
  { name: 'adjectiveNounPair', patterns: [ 'ADJ NOUN' ] }
];
nlp.learnCustomEntities( patterns );

This returns the pairs: 'beautiful friend', 'small cottage', 'fluffy cats', and _'playful dogs'. Now, if you want to extract only adjectives used for cats or dogs, you can do that using mark:

const patterns = [
  { name: 'adjectiveAnimalPair', patterns: [ '[ADJ] [cats|dogs]' ], mark: [0, 0] }
];
nlp.learnCustomEntities( patterns );

This returns: 'fluffy', 'playful'.

Shorthand patterns

Let us say we wish to extract noun phrases from the following text:

const text = 'Each time we gather to inaugurate a President we bear witness to the enduring strength of our Constitution.';

To keep things simple, we assume that a noun phrase could be simply composed as:

const patterns = [
  {
    name: 'nounPhrase',
    patterns: [
      'NOUN',
      'PROPN',
      'DET NOUN',
      'DET PROPN',
      'ADJ NOUN',
      'ADJ PROPN',
      'DET ADJ NOUN',
      'DET ADJ PROPN'
    ]
  }
];

The above list could be transformed into the following shorthand pattern:

const patterns = [
  { name: 'nounPhrase', patterns: [ '[|DET] [|ADJ] [NOUN|PROPN]' ] }
];

In shorthand patterns:

1. Options are listed between opening and closing square brackets
2. Each option is separated by a vertical pipe character as in [NOUN|PROPN]
3. There can not be any space character within the square brackets
4. An option may be empty as in the case of the first two sets of options — [|DET] and [|ADJ]
5. Between successive options lists, there should be one or more spaces — [|DET] [|ADJ] [NOUN|PROPN]
6. All options are automatically generated by finding all possible combinations — here there are 2-options in every list resulting in 2x2x2 = 8 combinations as shown in the previous pattern.

Given below is the complete code along with the output:

const text = 'Each time we gather to inaugurate a President we bear witness to the enduring strength of our Constitution.';
const patterns = [
  { name: 'nounPhrase', patterns: [ '[|DET] [|ADJ] [NOUN|PROPN]' ] }
];
nlp.learnCustomEntities( patterns );
const doc = nlp.readDoc( text );

doc.customEntities().out();
// -> [ 'Each time',
//      'a President',
//      'witness',
//      'the enduring strength',
//      'Constitution' ]

Escaping

If you need to match with literal ‘in the month of January’ instead of entity type DATE, then it must be escaped using a caret sign. This means ^January will match with literal token value January and not as a DATE entity. Similarly, in order to match literally with a part-of-speech tag like NOUN or VERB, you need to escape it too.

config

There are three parameters on which the match is performed. They are matchValue, usePOS and useEntity. These are of boolean type, which enables the custom entity recognizer to look for patterns in a sequence. The default configuration is matchValue: false, useEntity: true, usePOS: true.

During detection phase, match is performed in the following sequence:

1. Entity types are matched first
2. In case of no match at step 1, token value match is attempted
3. If no match is found at step 2, then at last the token’s part-of-speech is matched

To understand how the patterns are defined, we take an example of pizza ordering intent:

I wish to order 1 small classic with corn topping and 2 large supreme with Olives, Onion topping.

The table below shows examples of patterns that can be made to extract custom entities from this text. These patterns consist of either a single token, a phrase having multiple words (i.e. tokens), their entity type, part-of-speech, or any combination of these. Each pattern is matched at the token level.

These combinations serve as a starting point to learn the required patterns for custom entity detection. Let’s look at the following pizza intent example to understand the learning process. Custom entities can be defined as an array of objects with patterns for learning.

Example:

const text = 'I wish to order 1 small classic with corn topping and 2 large supreme with Olives, Onion topping.';
const pizza = [
  { name: 'Category', patterns: [ '[Classic|Supreme|Extravaganza|Favorite]' ] },
  { name: 'Qty', patterns: [ 'CARDINAL' ] },
  { name: 'Topping', patterns: [ '[Corn|Capsicum|Onion|Peppers|Cheese|Jalapenos|Olives]' ] },
  { name: 'Size', patterns: [ '[Small|Medium|Large|Chairman|Wedge]' ] }
 ];

nlp.learnCustomEntities( pizza, {
  matchValue: false,
  usePOS: true,
  useEntity: true
} );