Patterns Are Power: A Beginner’s Guide To Spam Detection

We’ve all gotten spam, we’ve all felt its hideous, annoying burden, and we’ve all (but Lord I hope not) had to lovingly explain to a grandparent that “No, that strange digital letter isn’t actually from a foreign king in need.” A bit played out, I know, but a poignant example, and one that highlights a key to understanding at least one of the ways we can avoid it altogether — but more on that later.

Photo by Lorie Shaull, sourced by Wikimedia Commons

Spam Detection Methods

If you do any amount of research, you’ll see that there are countless — and I barely mean that hyperbolically — ways to filter for spam, or just filter email in general. The approaches are as plentiful as there are types of phishy emails one can receive. There are networking-oriented solutions like tarpits, which selectively slow down suspected spam and allow ham messages (non-spam) to flow free. There are content-oriented solutions that check the URL’s in a given message against a database of URL’s that are commonly used for phishing or other malicious practices. And there are even solutions which depend on real-world action, such as reporting the IP from which spam originates and terminating the user’s internet service provision.

But, of course, there are other methods. And the key method I want to talk about, which was alluded to above, has to deal more with pattern recognition than anything else. I’m talking about machine learning.

Remember the grandparent example with the fake-royalty phishing scheme? Remember how you have either received an email like this or heard of someone who has? These two questions can only net agreement if there are generalizable patterns found in spam. Of course, this won’t always hold, but in many cases the language and/or syntax which spam tends to take can be meaningfully different from the nuanced communication made by human users. If this is true (surprise, it often is), then we can train a machine-learning model to tell the difference. This, my friends, is a binary classification problem.

Spam Detection And Binary Classification

For the uninitiated, binary classification in this context just means “Using machine learning to have the computer tell us what group something is in.” This logic extends to any number of categories, but here we just have the two. Before the theory rambling goes on and on, let’s get into a concrete example.

First, some samples of spam messages:

“URGENT! Your Mobile number has been awarded…”
“Loan for any purpose £500 — £75,000…”
“Congrats! Nokia 3650 video camera phone is yours…”

And some samples of ham messages:

“Hi Steven, your dropbox is full and stopped syncing…”
“Hope you’ve been well. I attached a photo of…”
“Thanks for responding! I have added you as a writer…”

There’s some patterns that pop out at first. The spam messages are typically asking something of us, or offering us something. Their purpose in not communication. Their purpose is engagement, typically monetary, that will net the spam-senders our personal information. But, that’s a fairly high-level semantic distinction to make. A simple neural network to be trained for the sake of example can’t pick out and identify these themes.

What it can instead pick out is word orders. Of course this is handled numerically, and it’s handle just like this: We take our dataset of spam and ham emails, fit a Tokenizer on the data, turn the data into sequences of numbers, and then pad the sequences to a uniform length. In Python code, this looks like:

We instantiate a Keras Tokenizer:

tokenizer = Tokenizer()

We fit it on the emails we have. Here it is in the form of a Pandas Series:

tokenizer.fit_on_texts(dataframe[‘email_message’])

We convert that same Series of emails to a bunch of sequences:

email_sequences = tokenizer.texts_to_sequences(dataframe[‘email_message’])

We convert them to a NumPy Array (this part is for compatibility with our neural network):

email_array = np.array(email_sequences)

We pad these sequences to a uniform length. In common language: We take a message that is, for example, 25 words long (and therefore a sequence of 25 numbers now) and add 75 zeroes in front. The pad_sequences() function comes from keras.preprocessing.sequence, by the way.

padded_email_array = pad_sequences(email_array, maxlen=100)

An example of an email before an after this process could look like:

‘REMINDER FROM O2: To get 2.50 pounds free call credit and details of great offers pls reply 2 this text with your valid name, house no and postcode’

Turning into:

[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 1341 42 510 2 30 17 255 511 46 15 836 9 322
18 109 632 69 116 17 41 99 31 13 368 254 228 37
9 1342]

With all of our spam and ham data processed as such, we can feed it into a neural network, and train it to separate the one group from the other. Note: There are a few more steps both before and after this, that are viewable here in the full notebook for this example. I am simplifying some things for the sake of explanation — focusing on the concept of relating word order to numerical sequence.

Spam Detection Neural Network

There are numerous ways to do this, but for the current workflow, I’m using a Sequential model in Keras. We are less interested here in the code, which is fully explained here, than in the outcomes. For transparency sake, the code is:

# Instantiate model
base_model = Sequential()
# Add model layers based on above rationale
base_model.add(Embedding(len(total_vocab), 100))
base_model.add(LSTM(8, return_sequences=True))
base_model.add(GlobalMaxPool1D())
base_model.add(Dense(25, activation=’tanh’))
base_model.add(Dense(2, activation=’softmax’))
# Compile model
base_model.compile(loss=’categorical_crossentropy’,
optimizer=’Adam’,
metrics=[‘accuracy’])
# Fit model and store results
base_results = base_model.fit(X_train, y_train,
epochs=20, batch_size=30,
validation_split=0.2)

What’s key here is the Dense layer with 2 nodes in it, which effectively is the final decider on whether or not “spam” or “ham” is the final decision made. Performance-wise, this model did as follows:

The neural network was able to use these numerical sequences to predict whether or not an email was spam with a 97.92% accuracy. That is exceptionally high, and only a piece of the puzzle. In practice, if you were developing a new spam-filtering technology, you would likely be using this machine-learning methodology in conjunction with some of the others mentioned in the introductory text. I encourage you to try this brief example out yourself, following along the same notebook linked here.

Reach out to me on Twitter @zych_steven with any feedback!