Quick Connections for MongoDB with Node-RED

After last week’s blog post where we connected an Internet of Things temperature sensor to record data in MongoHQ’s MongoDB instances, one Twitter follower commented that it would have been even easier if we had used Node-RED. We were going to write about Node-RED anyway but were keeping that previous post to the bare essentials.

Node-RED is an application that was originally developed for handling the MQTT protocol messages but is far from limited to that. With Node-RED, the processing of messages is visualised as a graphical flow between “nodes” representing input, output or processing by Node-RED. A message is generated at an input node and is passed to any node that node’s out is connected to. Built on top of the Node.js platform, it is able to also handle connections to databases such as MongoDB, protocols such as TCP, UDP, HTTP and WebSockets and services like IRC or Twitter and has nodes that represent input or output to all of them. It also has nodes which allow you to change values in a message, process a messages with JavaScript, fill out a template, delay a message being passed on, select a path for a message to take, map numeric ranges or even do sentiment analysis on text. There’s also some handy file storage nodes and, to aid debugging, a message injector node and a debug output node.

Things like this are best demonstrated though. Recently I was asked to set up a database sample of Twitter tweets about the floods in the UK and I decided to use Node-RED. If you want to learn how to install Node-RED in detail consult our how-to page. Suffice to say that once installed, to begin working with a running Node-RED, we just need to point our browser at port 1880 on the host it’s running on. Once there we get to see the Node-RED user interface.

Node-RED's user interface
Node-RED’s user interface

On the left is a palette of nodes we can drag into the workspace in the centre. On the right are two tabs, one which shows information and help about the currently selected node (either in the palette or in the workspace) and another for displaying debugging information. For our Flood tweets, we scroll drown and drag a Twitter input into the workspace. The red Deploy button lights up to show we’ve made changes we haven’t deployed yet. We can now double click on the dropped node to set it up.

Configuring the incoming Twitter node
Configuring the incoming Twitter node

This lets us select Twitter credentials (and handles authorisation with Twitter), select what feed we want to search (public, followers or direct messages) and enter what we want to search for. I’ve put in “#flood” for this example. There’s also a name field to give the node its own name. How will we know thats working? If we drag a debug node onto the workspace and then drag a connection from our Twitter node to the debug node, we’ll be able to see what is being generated. To start it running, click the Deploy button. After a while – you’ll be waiting for someone to tweet – the debug window should start showing the messages being generated.

Configuring the MongoDB out node
Configuring the MongoDB out node

Now to get those messages to MongoDB – we drag a MongoDB out node onto the workspace and double click that to configure it. We can configure a MongoDB server by selecting “add new mongodb” where we can enter the host, port, database and, if needed, username and password. We can also name the configuration. But there was a catch… MongoHQ requires authenticated clients but, as we were writing this, the released Node-RED didn’t have support for username and password authentication. That, by the way would have stopped us using it for last week’s Internet of Things project. When we mentioned this to the developers on Twitter they pushed support for authentication into the development version of Node-RED within an hour so everyone will be able to make use of it – our page on setting up Node-RED shows you how to get and install the development version so you can use Node-RED with MongoHQ’s sandboxes and production servers now. We can now set up our MongoDB node to save the incoming messages to a floodtweetsarchive collection in our database.

Once the database is set up, we can set the collection the message will be saved in, whether it should be saved, inserted or used to remove a document, and we can, if we want, limit it to the payload field. As the Twitter in node populates the message with all the Twitter information and we want to save that, we’ll not set that checkbox. We can now connect the Twitter in node to the MongoDB out node, leaving the debug connection in place and click Deploy.

Node-RED Twitter to MongoDB flow
A Twitter to MongoDB flow

We can then hop over to the MongoHQ admin console and look at the database and find a tweet already saved in the database. It’s that simple. The Node-RED server will continue to run the flow even after we’ve browsed away. This is probably the simplest Node-RED setup we could come up with. Say we needed something which did a some more processing, where we want to remove a lot of the extraneous data from the tweets and perform some evaluation of the text. Implementing this as a flow gives us…

A Node-RED flow with sentiment analysis and JavaScript functions
A flow with sentiment analysis and JavaScript functions

… where the sentiment node adds does analysis to the message based on parsing its payload text and a JavaScript function does the work of stripping out the tweet’s raw data, leaving only screen-name and friend/follower count. It’s that reduced data that gets written to MongoDB.

We started with the comment about our previous article and how we could use Node-RED to set up this functionality. To implement it, we would place an MQTT in node on the workspace and configure it to listen to a particular topic, then link it to a MongoDB out node. One problem there though is that we wouldn’t be able to do the single document with time series that we created with our Node only implementation because there is, at least currently, no update operation in the MongoDB nodes. This though is a good time to point out that you can add custom nodes and create your own nodes to Node-RED; they are made up of a HTML definition for the configuration page and a JavaScript implementation of the in and out functions.

Although Node-RED is a great tool for prototyping ideas, it is important to remember it is still a version 0.5 and as such omits things which would be essential for full-scale production like authentication and authorisation for the web front end and a separation of the run-time functionality from that front end. But development has been rapid and, as an open source project, Node-RED will see these features sooner rather than later. Right now though, it offers a powerful way of graphically wiring the world of data flows together, from social media and the Internet of Things or from more traditional sources, and populating your MongoDB databases in novel ways.

Written by Dj Walker-Morgan

Content Curator at Compose, Dj has been both a developer and writer since Apples came in ][ flavors and Commodores had Pets.