Discord Bot Part 3 - Using Azure Table Storage

In my previous post we modified our Discord bot to take messages generated in chat, put them on a queue and processed them using an Azure Function. In this post we are going to modify the function that reads the message to place the message in an additional storage queue. An additional Azure Function will monitor this queue and write the message to table storage. Thankfully we have our core worker in place, this post, and future posts, should be a little bit shorter as we are now making incremental changes and improvements.

Cleaning up some naming

Before we start adding an additional queue into our code we are going to quickly change the existing queue in our main discord bot code to make it a little easier to read and so we don’t mix up our queues. In our DiscordSocketService.cs file we are changing the variable inboundQueue to discordMessagesQueue. This variable is used in a few places at the moment, for instance at the top where variables are declared in the DiscordSocketService.cs file.

This makes our code a little bit easier to read, but we also want to make sure we can distinguish our queue’s when look at them in the storage accounts. To do that let’s change the name of the queue that stores these messages from discord-bot-inbound-queue to discord-messages-inbound-queue. The first place we need to change that is in the ConfigureStorageQueue() method, as shown here:

discordMessagesQueue = queueClient.GetQueueReference("discord-messages-inbound-queue");

We also need to make sure we update the Azure Function that process this queue in the Functions project. The queue is currently referenced in the signature of the InboundMessageProcess and should be changed to look like this:

public static string InboundMessageProcess([QueueTrigger("discord-messages-inbound-queue")] CloudQueueMessage myQueueItem, ILogger log)

Now we are being pretty specific with our naming, let’s add an additional queue to start persisting data to Azure Table Storage.

Adding another queue and function

Let’s add another queue to push the message to so that we can store the message to Table Storage. The reason we are using a separate queue and function is to maintain service autonomy and loose coupling. Firstly, let’s create a new CloudQueue variable at the top of the DiscordSocketService:

private CloudQueue discordActivityQueue;

Once this variable has been created, we need to initialize it in the ConfigureStorageQueue() method. Add two additional lines to the bottom of this method to initialize the queue the same way as the message queue:

discordActivityQueue = queueClient.GetQueueReference("discord-activity-inbound-queue");
discordActivityQueue.CreateIfNotExistsAsync();

We also need to add the message onto the queue when a user creates a message the same way as before, so add an additional line in the RecieveMessage method:

discordActivityQueue.AddMessage(jsonMessage);

This will leave us with a Discord bot that is now pushing duplicate messages to two different queues. We’ll leave the original alone because that should still be working fine we know need to add a new Azure Function to grab messages from the second queue and persist them into Table Storage. If we run the code now our bot will still respond to the !ping command, however all messages will now also get pushed into an audit queue for us to process.

Creating table entities

Before we create a Function to retrieve messages off the queue, lets have a look at how we are going to store our messages for later. For this walk through we are going to use Azure table storage as it allows us to store a lot of independent data quickly and easily without having to worry about managing a database. Table storage requires two pieces of data: a partition key and a row key. Essentially a partition key is how we segment groups of data and a row key is a unique identifier for that data.

Ideally you would plan out the data you are storing but for our use case we are not going to worry to much about a strategy for Partitioning Table Storage. In this instance we will just use the channel identifier for the partition key and the message ID generated by Discord for the row key. So let’s first add the message ID to our converted message entity. At the top of the ConvertedMessage entity in our DNetUtils class library add the variable:

public ulong MessageId { get; set; }

And in the constructor for ConvertedMessage add the following line to initialize the variable:

MessageId = message.Id;

With this change added in, let’s also create a new entity that represents the data we are storing to Table Storage. Create a folder in your utils class library called TableEntities and we are going to create a new class in their called ReceivedMessage. This class will have the following properties:

public string PartitionKey { get; set; }        // Channel ID
public string RowKey { get; set; }              // Message ID
public string AuthorId { get; set; }
public int Source { get; set; }                 // System, User, Bot, Webhook
public string Content { get; set; }
public DateTime CreatedAt { get; set; }

There are a few differences here worth going through based on what is in the ConvertedMessage object. Firstly, Table Storage only contains a limited set of data types so most of the data will need to be converted back to basic types. Secondly, we have no way of storing the array of values that contain various mention ID’s. Because we are not doing anything with these this is not a big issue so I’ve just left them off. We are going to create a constructor for this class which takes a ConvertedMessage as input and converts it two our table entity. This looks like:

public ReceivedMessage(ConvertedMessage message)
{
    PartitionKey = message.ChannelId.ToString();
    RowKey = message.MessageId.ToString();
    AuthorId = message.AuthorId.ToString();
    Source = message.Source.GetHashCode();
    Content = message.Content;
    CreatedAt = message.CreatedAt.UtcDateTime;
}

This will then allow us to take our message off the queue and store it in Table storage with very minimal code in the Function.

Process the activity queue

Now we have a data type to convert to we can now create a really small, simple function to store data to Table Storage. We will reuse all the existing storage settings so that messages get stored in the same account as the Storage queues. Add a new Function (I’ve called me ProcessActivity to our functions project, select it as a queue trigger and leave the details blank, we will edit this next anyway.

The code for this class is fairly simple now:

[FunctionName("InboundActivityProcess")]
[return: Table("ActivityTable")]
public static ReceivedMessage InboundActivityProcess([QueueTrigger("discord-activity-inbound-queue")] CloudQueueMessage myQueueItem, ILogger log)
{
    log.LogInformation($"Activity Queue trigger function processed: {myQueueItem}");

    ConvertedMessage message = DiscordConvert.DeSerializeObject(myQueueItem.AsString);
    return new ReceivedMessage(message);
}

This code is pretty similar to our existing Function, with the return line changed to the table labeled ActivityTable. The signature of the method has also changed to monitor the queue named discord-activity-inbound-queue. The two lines within this method: deserialize the message received from the queue and; convert the message to the table entity type and send it to the ActivityTable table.

If we run our bot now and start entering messages you will see our functions saving the messages to Table storage.

What next

Our code now keeps messages generated within our channel to table storage. Next time we are going to look at integrating some more advanced services with Azure Cognitive Services to do some more complex work, such as sentiment analysis or spell check.