"# UFCFEL-15-3 Security Data Analytics and Visualisation\n",
"# Portfolio Assignment 3: Large-Scale Data Exploration for Insider Threat Detection (2022)\n",
"---\n",
"\n",
"The completion of this worksheet is worth a **maximum of 45 marks** towards your portfolio assignment for the UFCFEL-15-3 Security Data Analytics and Visualisation (SDAV) module.\n",
"\n",
"### Brief\n",
"---\n",
"\n",
"In this task, you have been asked to investigate a potential security threat within an organisation. Building on your previous worksheet expertise, you will need to apply your skills and knowledge of data analytics and visualisation to examine and explore the datasets methodically to uncover which employee is acting as a threat and why. The company have provided you with activity logs for various user interactions for the past 6 months, resulting in a lot of data that they need your expertise for to decipher. They want to have a report that details the investigation that you have carried out, details of the suspected individual, and a clear rationale as to why this suspect is flagged. You will need to document your investigation, giving clear justification for your process using Markdown annotation within your notebook. You will need to provide a clear rationale for why you suspect a given individual to be acting as a threat, based on the pattern of activity that you identify.\n",
"\n",
"<i>This coursework is specifically designed to challenge your critical thinking and creativity, and is designed as an open problem. Examine the data and try to think how an individual user may appear as an anomaly against the remainder of the data. This could be an anomaly compared to a group of users, or an anomaly as compared over time.</i>\n",
"\n",
"### Assessment and Marking\n",
"---\n",
"\n",
"Marks will be allocated within the following criteria:\n",
"\n",
"* **Identification and justification of the suspicious behaviour (15)**\n",
"* **Analytical process and reasoning to deduce the suspicious behaviour (15)**\n",
"* **Use of informative visualisation and data exploration techniques (10)**\n",
"* **Clarity and professional presentation (5)**\n",
"\n",
"To achieve the higher end of the grade scale, you need to demonstrate creativity in how you approach the problem of identifying malicious behaviours, and ensure that you have accounted for multiple anomalies across the set of data available.\n",
"\n",
"This assignment should be submitted as as PDF to your Blackboard portfolio submission as per the instructions in the assignment specification available on Blackboard. A copy of your work should also be provided via a UWE Gitlab repository, with an accessible link provided with your portfolio.\n",
"\n",
"### Contact\n",
"---\n",
"\n",
"Questions about this assignment should be directed to your module leader (Phil.Legg@uwe.ac.uk). You can use the Blackboard Q&A feature to ask questions related to this module and this assignment, as well as the on-site teaching sessions.\n",
"The cell above is creating a set of DataFrames to work with. The set of tables are named as follows:\n",
"\n",
"* employee_data\n",
"* login_data\n",
"* usb_data\n",
"* web_data\n",
"* file_data\n",
"* email_data\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 1. Begin investigation\n",
"\n",
"To start I will investigate exactly who has been attempting to access folders related to security and then crossreference this with the employee data to see if anyone outside of the security roles has been attempting to access data they should not have access to."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>user</th>\n",
" <th>role</th>\n",
" <th>email</th>\n",
" <th>pc</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"Empty DataFrame\n",
"Columns: [user, role, email, pc]\n",
"Index: []"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# ANSWER\n",
"#In order to find anyone attempting to access a folder they should not be I created a filter which checks which users have attempted to access \"Security\" files.\n",
"#In the end no users without the security or technical roles had attempted to access the folders."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"for the next step of my investigation I decided to check who was logging on at suspicious times this would give me a good starting point for the investigation by flagging up certain users. I started this task by taking the general average across the entire dataset of how many people were logging in from the hours of 12AM-6AM, assuming that these are not working hours. A spike in activity could imply that malicious activity was being performed around those hours, a drop in activity could be the direct result of someone acting maliciously (for example taking down login servers)."
"The above data shows all of the dates which have a mean number of logins which is either 10 above or below the mean. This has narrowed down my investigation to 24 days as opposed to six months. To continue the investigation I will check the specific dates with a higher number of logins to see if there's anyone consistently causing them. In the end I was unable to get this to work however I believe that if the information was properly checked there would be a single point of failure which could be used for further analysis."
"The unusually high levels of activity listed above imply that there is some sort of malicious activity going on on those dates. Further investigation, specifically of who exactly was logging in on those dates is likely to reveal the party responsible for these discrepencies."
# UFCFEL-15-3 Security Data Analytics and Visualisation
# Portfolio Assignment 3: Large-Scale Data Exploration for Insider Threat Detection (2022)
---
The completion of this worksheet is worth a **maximum of 45 marks** towards your portfolio assignment for the UFCFEL-15-3 Security Data Analytics and Visualisation (SDAV) module.
### Brief
---
In this task, you have been asked to investigate a potential security threat within an organisation. Building on your previous worksheet expertise, you will need to apply your skills and knowledge of data analytics and visualisation to examine and explore the datasets methodically to uncover which employee is acting as a threat and why. The company have provided you with activity logs for various user interactions for the past 6 months, resulting in a lot of data that they need your expertise for to decipher. They want to have a report that details the investigation that you have carried out, details of the suspected individual, and a clear rationale as to why this suspect is flagged. You will need to document your investigation, giving clear justification for your process using Markdown annotation within your notebook. You will need to provide a clear rationale for why you suspect a given individual to be acting as a threat, based on the pattern of activity that you identify.
<i>This coursework is specifically designed to challenge your critical thinking and creativity, and is designed as an open problem. Examine the data and try to think how an individual user may appear as an anomaly against the remainder of the data. This could be an anomaly compared to a group of users, or an anomaly as compared over time.</i>
### Assessment and Marking
---
Marks will be allocated within the following criteria:
***Identification and justification of the suspicious behaviour (15)**
***Analytical process and reasoning to deduce the suspicious behaviour (15)**
***Use of informative visualisation and data exploration techniques (10)**
***Clarity and professional presentation (5)**
To achieve the higher end of the grade scale, you need to demonstrate creativity in how you approach the problem of identifying malicious behaviours, and ensure that you have accounted for multiple anomalies across the set of data available.
This assignment should be submitted as as PDF to your Blackboard portfolio submission as per the instructions in the assignment specification available on Blackboard. A copy of your work should also be provided via a UWE Gitlab repository, with an accessible link provided with your portfolio.
### Contact
---
Questions about this assignment should be directed to your module leader (Phil.Legg@uwe.ac.uk). You can use the Blackboard Q&A feature to ask questions related to this module and this assignment, as well as the on-site teaching sessions.
---
%% Cell type:markdown id: tags:
## Load in the data
%% Cell type:code id: tags:
``` python
# DO NOT MODIFY THIS CELL - this cell is splitting the data to provide a suitable subset of data to work with for this task.
# If you change this cell your output will differ from that expected and could impact your mark.
The cell above is creating a set of DataFrames to work with. The set of tables are named as follows:
* employee_data
* login_data
* usb_data
* web_data
* file_data
* email_data
%% Cell type:markdown id: tags:
# 1. Begin investigation
To start I will investigate exactly who has been attempting to access folders related to security and then crossreference this with the employee data to see if anyone outside of the security roles has been attempting to access data they should not have access to.
%% Cell type:code id: tags:
``` python
# ANSWER
#In order to find anyone attempting to access a folder they should not be I created a filter which checks which users have attempted to access "Security" files.
#In the end no users without the security or technical roles had attempted to access the folders.
```
%% Output
Empty DataFrame
Columns: [user, role, email, pc]
Index: []
%% Cell type:markdown id: tags:
for the next step of my investigation I decided to check who was logging on at suspicious times this would give me a good starting point for the investigation by flagging up certain users. I started this task by taking the general average across the entire dataset of how many people were logging in from the hours of 12AM-6AM, assuming that these are not working hours. A spike in activity could imply that malicious activity was being performed around those hours, a drop in activity could be the direct result of someone acting maliciously (for example taking down login servers).
The above data shows all of the dates which have a mean number of logins which is either 10 above or below the mean. This has narrowed down my investigation to 24 days as opposed to six months. To continue the investigation I will check the specific dates with a higher number of logins to see if there's anyone consistently causing them. In the end I was unable to get this to work however I believe that if the information was properly checked there would be a single point of failure which could be used for further analysis.
The unusually high levels of activity listed above imply that there is some sort of malicious activity going on on those dates. Further investigation, specifically of who exactly was logging in on those dates is likely to reveal the party responsible for these discrepencies.
