Using Azure Text Analysis API to extract sentiment and key phrases from text

Do analyst biases affect the relationship between the narrative sentiment in their reports and the recommendation and target price outputs produced for a given stock and are analysts hedging their recommendations with the narrative?

Below is a statement made by Irving Fisher, a famous economist, seven day before The Wall Street Crash of 1929. The quote generated a sentiment score considered negative. So, was he biased? Did he believe the statement he was making? If he did, it must be the worst stock tip in history. Along with the sentiment, the key words and phrases are also extracted from the quote to help identify the main points.

How did I calculate the sentiment score and extract key phrases from the quote? In this post I am going to tell you. I utilized Microsoft Azure Text Analytics API, a cloud-based service that provides advanced natural language processing over raw text, and includes four main functions:

  • Sentiment analysis: This API returns a sentiment score between 0 and 1 for each document, where 1 is the most positive.
  • Key phrase extraction: Extracts key phrases to identify the main points.
  • Language detection: Detects which language the input text is written in, up to 120 languages.
  • Entity linking: Identifies well-known entities in your text and links to more information on the web.

In this post, we will look at sentiment analysis and key phrase extraction, which is part of Cognitive Services in the Azure portal. To get started, open the Azure portal and locate AI and Cognitive Services, then the Text Analysis API, open and fill out the form. Be sure to select the Free tier (F0) as shown below:

Next copy the value of key 1 for future use.

For this post, I am going to use ASP.NET Web API to call the Text Analytics API and store the data in a SQL Server table to be accessed by an Angular component to render Irving’s results as shown above. Before we get started with the code, let’s define a few classes that we need for processing.

// Input

public class TextSentiment

{

  decimal id { get; set; }

  public string language { get; set; }

  public string text { get; set; }

}

public class TextSetimentInput

{

  public List<TextSentiment> documents { get; set; }

}

// Results

public class TextErrorResult

{

  public decimal id { get; set; }

  public string message { get; set; }

}

public class TextSetimentResult

{

  public decimal id { get; set; }

  public decimal? score { get; set; }

  public string keyPhrasesString { get; set; }

  public List<string> keyPhrases { get; set; }

}

public class TextSetimentResults

{

  public List<TextSetimentResult> documents { get; set; }

  public List<TextErrorResult> errors { get; set; }

}


The method below LoadedTextSentiment() does the following:

  • Creates the Text Analysis Input object _tsStrings with the desired string or strings to be analyzed .
  • Gets the sentiment score of the text input by creating the restClient and call PostTextAnalyticsAsync passing the input data (_tsStrings) and “sentiment”.
  • Gets the key phrases of the text input by creating the restClient2 and call PostTextAnalyticsAsync passing the input data (_tsStrings) and “keyPhrases”.
  • Combines the results and call stored procedure to insert into a table.

public async void LoadTextSentiment()

{

    try

    {

    TextSetimentResults _tsResults = new TextSetimentResults();

    List<TextSentiment> items = new List<TextSentiment>();

    TextSetimentInput _tsStrings = new TextSetimentInput();

    _tsStrings.documents = new List<TextSentiment>();

    _tsResults.documents = new List<TextSetimentResult>();

   

    // Create input object

    string _text = "Stock prices have reached what looks like a permanently high plateau. I do not feel there will be soon if ever a 50 or 60 point break from present levels, such as (bears) have predicted. I expect to see the stock market a good deal higher within a few months.";

    items.Add(new TextSentiment { id = 99999999, language = "en", text =_text });

    _tsStrings.documents = items;

   

    // Call post to get sentiment

    var restClient = new

    Http.RestClient<TextSetimentResults>(null, new Http.LoggingHandler(new HttpClientHandler()));

    var _result = await restClient.PostTextAnalyticsAsync(_tsStrings, "sentiment");

    if (_result != null && _result.documents != null)

    {

      // Call post to get keywords

      var restClient2 = new Http.RestClient<TextSetimentResults>(null, new Http.LoggingHandler(new HttpClientHandler()));

      var _result2 = await restClient2.PostTextAnalyticsAsync(_tsStrings, "keyPhrases");

      // loop through input

      foreach (var item in _result.documents)

      {

         TextSetimentResult _tsResult = new TextSetimentResult() { id = item.id, score = item.score };

        

         if (_result2 != null && _result2.documents != null)

         {

           string kps= null;

           var _kpresults= _result2.documents.Where(x => x.id == item.id).FirstOrDefault();

           if (_kpresults != null)

           {

             kps = String.Join(",",kpresults.keyPhrases.Select(x => x.ToString()).ToArray());

           }

           _tsResult.keyPhrasesString = kps;

         }

       }

         // Insert into table

       await Db.Database.ExecuteSqlCommandAsync("InsertTextAnalytics Id,Sentiment, KeyPhrases",

                           new SqlParameter("RecommendId", _tsResult.id),

                           new SqlParameter("Sentiment", _tsResult.score),

                           new SqlParameter("KeyPhrases", _tsResult.keyPhrasesString));

                             _tsResults.documents.Add(_tsResult);

      }

    }

    }

    catch (Exception ex)

    {

      var error = ex;

    }

    return;

}


The method PostTextAnalyticsAsync shown below takes as arguments the input data and the call type (either “sentiment” or “keyphrases”) and makes the appropriate post request and returns the results. The base address for the post request is https://eastus.api.cognitive.microsoft.com/text/analytics/v2.0/ plus the call type. You need to set the appropriate headers including

  • Ocp-Apim-Subscription-Key = Should be your Key 1 (from above).
  • Content-Type = Set it to application/json.
  • Accept = Set it to application/json.

public async Task<TResult> PostTextAnalyticsAsync<TData>(TData documents, string calltype)

{

 TResult result = default(TResult);

 string _subscriptionKey = "xxxxxxxxxxxx";

 string _taurl = "https://eastus.api.cognitive.microsoft.com/text/analytics/v2.0/";

 try

 {

    Dictionary<string, IEnumerable<string>> _headers = new Dictionary<string,  IEnumerable<string>>();

    using (var client = new HttpClient(Handler))

    {

        client.DefaultRequestHeaders.Accept.Clear();

        client.Timeout = TimeSpan.FromMinutes(10);

        client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));

        client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", _subscriptionKey);

 

        if (_headers.Count > 0)

        {

          foreach (var key in _headers.Keys)

          {

            client.DefaultRequestHeaders.Add(key, _headers[key]);

            }

        }

        var response = await client.PostAsync(_taurl + calltype, documents, _formatter);

        result = await ReadSingleResult(response);

    }

 }

 catch (Exception ex)

 {

    var error = ex;

 }

 return result;

}


The SQLServer table ddl and the stored procedure that I used to store the text analysis data is below.

 

CREATE TABLE [dbo].[TextAnalytic](

       [Id] [numeric](18, 0) NOT NULL,

       [sentiment] [numeric](19, 4) NULL,

       [keyPhrases] [nvarchar](1000) NULL,

       [CreatedBy] [nvarchar](500) NOT NULL,

       [CreatedDateTime] [datetime] NOT NULL,

       [UpdatedBy] [nvarchar](500) NOT NULL,

       [UpdatedDateTime] [datetime] NOT NULL,

CONSTRAINT [PK_TextAnalytic] PRIMARY KEY CLUSTERED

(

       [Id] ASC

) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)

)

 

 

CREATE PROCEDURE [dbo].[InsertTextAnalytics]   Id   numeric(18,0), Sentiment numeric(19,4), KeyPhrases nvarchar(1000)

AS

BEGIN

       SET NOCOUNT ON;

       INSERT INTO [dbo].[RecommendationTextAnalytic]

           ([Id]

           ,[sentiment]

           ,[keyPhrases]

           ,[CreatedBy]

           ,[CreatedDateTime]

           ,[UpdatedBy]

           ,[UpdatedDateTime])

     VALUES

           (Id , Sentiment, keyPhrases, 'azure',getdate(),'azure',getdate())

      

END

GO


The text analysis data is accessible through SQLSever can be rendered in a front-end application, such as a dashboard. Below is a simple angular component and html that renders the Irving Fisher panel above.

 

import { Component, Input } from 'angular/core';

 

Component({

    selector: 'sentiment-detail',

    templateUrl: 'sentiment-detail.component.html'

})

 

export class SentimentDetailComponent {

   Input('textSentimentOutput') textSentimentOutput: any;  

}

 

<div>

  <table   *ngIf='textSentimentOutput' class="table table-condensed table-borderless bg-primary">

  <tbody>

  <tr>

       <td>

           <img src="/content/images/fisher.jpg" class="img-thumbnail" />

       </td>

       <td>

           <blockquote class="blockquote">

             {{textSentimentOutput.text}}<br/><br/>{{textSentimentOutput.name}}

           </blockquote>

           <p class="bold">SENTIMENT:&nbsp;&nbsp;

             <span>{{textSentimentOutput.sentimentScore | number:'1.0-4'}}</span>      

             &nbsp;<span>{{textSentimentOutput.sentimentDesc}}</span>

           </p>

           <p class="bold">KEYWORDS:&nbsp;&nbsp;{{textSentimentOutput.keywords}} </p>

       </td>

  </tr>

  </tbody>

  </table>

</div>


In this post, I have described a method for analyze text to extract the sentiment score and key phrases utilizing Microsoft Azure Text Analytics API. At the beginning of the post, I suggested a possible use of this information for monitoring analyst biases through analyzing the narratives contained in their reports and by storing the results from the Text Analytics API. You can establish patterns by averaging the sentiment scores for the analyst and for the analyst and the stock being recommended to glean additional insight. As you can see, it’s very straight forward to get up and running with the Text Analytics API. The harder job is making sense of this information which I will leave to you. I hope you found this post useful.

Let's discuss your project .... Give us a call or drop us a line