Custom Score Engine
Oscova uses an in-built highly optimized performance and accuracy centric scoring engine to score stored intents by analyzing their expressions. Sometimes researchers and developers may want to plug-in their own custom implementations that can compute the similarity between a user message and an expression. For this Oscova exposes the IScoreEngine
interface via OscovaBot.ScoreEngine
property. Internally a default implementation is used but developers can replace the default scoring anytime prior to bot training.
The score engine has the following tasks to perform:
- Compute similarity between user request and expression tokens. The returned value must be between 0 and 1.
- Train on the provided collection of expression samples and return a model data if generated.
- Load the previously trained model data on deployment.
Note
Specifying a new implementation of IScoreEngine
interface does not stop the Internal Engine from computing the expression and request similarities. The internal engine still computes and returns the IntentResult
to the new implementation of the IScoreEngine
via the internalResult
parameter. It is then upto the new implementation to analyze the intent result in addition to the expression and request tokens.
The EvaluationSet
provides a collection of both expression and request tokens that comprises of the following:
- Tokenized Words
- Contexts prefixed with # sign
- Entities prefixed with @ sign
Custom Engine
The following implementation shows how one can replace the score engine to create a Command Bot that only works on discrete command like user inputs with no contextual settings. For brevity we will only implement the GetIntentScore
method of the interface.
public class CustomScoreEngine : IScoreEngine
{
public double GetIntentScore(EvaluationSet evaluationSet, IntentResult internalResult)
{
var expressionSample = evaluationSet.ExpressionSample;
var requestSample = evaluationSet.RequestSample;
//Token count does not match so return 0 similarity
if (expressionSample.Tokens.Count != requestSample.Tokens.Count) return 0d;
for (var i = 0; i < expressionSample.Tokens.Count; i++)
{
//Simply check if every expression sample token matches the request sample token.
if (expressionSample.Tokens[i] != requestSample.Tokens[i]) return 0d;
}
//Else return 100% similarity.
return 1;
}
public void Load(ModelData modelData)
{
//Do Nothing
}
public ModelData Train(DataSet trainingDataSet)
{
//Engine does not generate a model so just return an empty model data.
return ModelData.Empty;
}
}
In the above example we've created a new IScoreEngine
implementation that simply checks if the user request exactly matches the expression. If not then a value of 0 is returned.
Providing Trained Model
If your custom implementation of IScoreEngine
generates a trained model, you can return the model data or file in the Train()
method. This method is called when the bot training starts. When a non-empty data is received Oscova will include this data in the Unified Language Model file that is generated when the bot is fully trained.
public ModelData Train(DataSet trainingDataSet)
{
SomeAlgorithm.TrainOnData(trainingDataSet);
var fileStream = new FileStream(@"c:\trained\model.zip", FileAccess.Read);
return new ModelData(fileStream);
}
Loading Model Data
If your score engine generated a model during the bot training process and the Train()
method returned the model data as a stream. You can implement the Load()
method to process the stream that will be passed to the method on bot training.
public void Load(ModelData modelData)
{
SomeAlgorithm.LoadModel(modelData);
}