Hi!
Yesterday the Machine Learning.Net team announced a new version 0.4. There are several interesting novelties, however, I like the mention of Word Embedding. I have put to read a little about it and the truth is that the ability to use some existing models of text processing and on the same build our models is something that is appreciated.
In the original team post they shared tons of materials references about Word Embeddings, I have decided to take the sample Console App from the repository and test the differences between classic processing and what we can do with WE.
After a quick test, and using the same data Set for analysis and evaluation, the classic model with works with a Accuracy of 66.60% and using WE the Accuracy goes up to 72.30%
The code for the test is this one. The main difference is in the functions TrainModel() and TrainModelWordEmbeddings().
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| internal static class Program | |
| { | |
| private static PredictionModel<SentimentData, SentimentPrediction> _model; | |
| private static PredictionModel<SentimentData, SentimentPrediction> _modelWordEmbeddings; | |
| private static string AppPath => Path.GetDirectoryName(Environment.GetCommandLineArgs()[0]); | |
| private static string TrainDataPath => Path.Combine(AppPath, "datasets", "sentiment-imdb-train.txt"); | |
| private static string TestDataPath => Path.Combine(AppPath, "datasets", "sentiment-yelp-test.txt"); | |
| private static string ModelPath => Path.Combine(AppPath, "SentimentModel.zip"); | |
| private static void Main(string[] args) | |
| { | |
| TrainModel(); | |
| TrainModelWordEmbeddings(); | |
| Evaluate(_model, "normal"); | |
| Evaluate(_modelWordEmbeddings, "using WordEmbeddings"); | |
| Console.ReadLine(); | |
| } | |
| public static void TrainModel() | |
| { | |
| var pipeline = new LearningPipeline(); | |
| pipeline.Add(new TextLoader(TrainDataPath).CreateFrom<SentimentData>()); | |
| pipeline.Add(new TextFeaturizer("Features", "SentimentText")); | |
| pipeline.Add(new FastTreeBinaryClassifier() { NumLeaves = 5, NumTrees = 5, MinDocumentsInLeafs = 2 }); | |
| Console.WriteLine("=============== Training model ==============="); | |
| var model = pipeline.Train<SentimentData, SentimentPrediction>(); | |
| Console.WriteLine("=============== End training ==============="); | |
| _model = model; | |
| } | |
| public static void TrainModelWordEmbeddings() | |
| { | |
| var pipeline = new LearningPipeline(); | |
| pipeline.Add(new TextLoader(TrainDataPath).CreateFrom<SentimentData>()); | |
| pipeline.Add(new TextFeaturizer("FeaturesA", "SentimentText") { OutputTokens = true }); | |
| pipeline.Add(new WordEmbeddings(("FeaturesA_TransformedText", "FeaturesB"))); | |
| pipeline.Add(new ColumnConcatenator("Features", "FeaturesA", "FeaturesB")); | |
| pipeline.Add(new FastTreeBinaryClassifier() { NumLeaves = 5, NumTrees = 5, MinDocumentsInLeafs = 2 }); | |
| Console.WriteLine("=============== Training model with Word Embeddings ==============="); | |
| var model = pipeline.Train<SentimentData, SentimentPrediction>(); | |
| Console.WriteLine("=============== End training ==============="); | |
| _modelWordEmbeddings = model; | |
| } | |
| private static void Evaluate(PredictionModel<SentimentData, SentimentPrediction> model, string name) | |
| { | |
| var testData = new TextLoader(TestDataPath).CreateFrom<SentimentData>(); | |
| var evaluator = new BinaryClassificationEvaluator(); | |
| Console.WriteLine("=============== Evaluating model {0} ===============", name); | |
| var metrics = evaluator.Evaluate(model, testData); | |
| Console.WriteLine($"Accuracy: {metrics.Accuracy:P2}"); | |
| Console.WriteLine($"Auc: {metrics.Auc:P2}"); | |
| Console.WriteLine($"F1Score: {metrics.F1Score:P2}"); | |
| Console.WriteLine("=============== End evaluating ==============="); | |
| Console.WriteLine(); | |
| } | |
| } |
As I commented at the beginning the interesting thing about this new RElease is the ability to use various models pretrained. In the MSDN post we talk about GloVe, FastText and SSWE. The next step would be to see how the new models behave using some of these models
There are already several models, I’ll just do the test with Some of them because, during the training process, the existing models are downloaded Ondemand and download ~ 6GB per test is at least interesting
Well, the results are pretty interesting
=============== Evaluating model normal =============== Accuracy: 66.60% Auc: 73.97% F1Score: 61.78% =============== End evaluating =============== =============== Evaluating model using WordEmbeddings =============== Accuracy: 72.30% Auc: 81.19% F1Score: 70.50% =============== End evaluating =============== =============== Evaluating model using WordEmbeddings GloVe50D =============== Accuracy: 66.10% Auc: 69.32% F1Score: 64.28% =============== End evaluating =============== =============== Evaluating model using WordEmbeddings GloVe300D =============== Accuracy: 67.80% Auc: 73.23% F1Score: 66.60% =============== End evaluating =============== =============== Evaluating model using WordEmbeddings GloVeTwitter50D =============== Accuracy: 65.30% Auc: 70.06% F1Score: 64.26% =============== End evaluating =============== =============== Evaluating model using WordEmbeddings GloVeTwitter200D =============== Accuracy: 65.40% Auc: 72.63% F1Score: 64.69% =============== End evaluating =============== =============== Evaluating model using WordEmbeddings Sswe =============== Accuracy: 72.30% Auc: 81.19% F1Score: 70.50% =============== End evaluating ===============
El código completo de la app se puede descargar desde https://github.com/elbruno/Blog/tree/master/20180808%20MLNET%200.4%20WordEmbeddings
Happy Coding!
Greetings @ Toronto
El Bruno
References
My Posts
- Error ‘Entry point ‘Trainers.LightGbmClassifier’ not found’ and how to fix it
- Machine Learning Glossary of terms
- Export Machine Learning.Net models to ONNX format
- Loading Data In our Learning Pipeline With List (Lists for ever!)
- What’s new in version 0.2.0
- What’s a Machine Learning model? A 7 minute video as the best possible explanation
- Write and Load models using Machine Learning .Net
- Understanding the step by step of Hello World
- Hello World in ML.Net, Machine Learning for .Net !
El Bruno 
12 comments