diff --git a/.ipynb_checkpoints/UFCFVQ-15-M Programming Task 1 Template-checkpoint.ipynb b/.ipynb_checkpoints/UFCFVQ-15-M Programming Task 1 Template-checkpoint.ipynb
index 01f60e096d4ddb0f3a0e9abaf543f18874984944..f797f496952d066c2997368bf0b04f8ce8277b63 100644
--- a/.ipynb_checkpoints/UFCFVQ-15-M Programming Task 1 Template-checkpoint.ipynb
+++ b/.ipynb_checkpoints/UFCFVQ-15-M Programming Task 1 Template-checkpoint.ipynb
@@ -307,7 +307,17 @@
"any new line characters using .strip(). Next each item within all the rows using the index will be inserted into the the sublists of the empty data list.\n",
"Next I will create a dictionary to store all this data in and assign each sublist with the correct headings.\n",
"As good practive the CSV file should then be closed. Finally you will use the return command which will exit the function and allow the function to produce values based on the return key used.\n",
- "\n"
+ "\n",
+ "file_csv = \"usefile.csv\"\n",
+ "def file(file_csv):\n",
+ " file_open = open(file_csv, \"r\")\n",
+ " file_open.readline()\n",
+ " data = [[],[],[],[],[],[],[],[],[]]\n",
+ " for row in file_open:\n",
+ " items = row.strip().split(\",\")\n",
+ " items = [float(i) for i in items]\n",
+ " file_open.close()\n",
+ " return task"
]
},
{
@@ -505,10 +515,10 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "### FR1.1 - FR1.8\n",
- "These tasks set a good basis for the assignment while the previous worksheets helped with the understanding of lists and the different operators that could be used to solve the summary statistics. I placed FR1.8 at the top of the assignment as it was easier to use the data given then make up my own, opening this file was relatively easy with the main issues occurring when taking the list from strings to integers to enable me to perform different calculations on it. For the standard deviation the use of a loop made the code more concise and enabled me to create a variable and use the += assignment which adds a number to this variable until the loop gone through each item in the list. Furthermore, I also used the round operator which helped make the value easier to read which meant the table further below will looked pleasing. Once again throughout these questions there was a lot of use of the basic operators that Python offers which made the code more concise. When completing the questions for the percentiles I struggled with the indexes as I was uncertain if the value produced included the 0 index or if it did not. I decided to take 1 off these values produce to get it into index form as I felt this was the correct way to do this and then merely found this value in the sorted list. There was a great link on the internet that gave me a good idea how to create a nicely formatted table for FR1.7 which involved calling the answers from the previous tasks and meant that this table would work for new .dat files.\n",
- "### FR1.9 - FR1.12\n",
- "The tasks that involved task1.csv file it was challenging, with Fr1.10 causing issues and so I became creative in storing this csv to memory. I felt the best way to approach this task was to remove the header,create a loop which looked at each row, removing the comma and new line character. Then I created a loop for each value in the row and appended this to a nested list. Next, I created a dictionary with the headings to store each value. For the summary statistics I created a function that had empty dictionaries which would be filled with the summary statistics for each header. I was unable to get standard deviation working within these, as I couldn’t figure out how to create the sum((v-mean[h])^2) to work for each heading however I have left the code in but commented out. I wish I had made the function print clearer results however this task did cause me to think outside the box and create a more complex function. For FR1.12 I followed the same link as for FR1.7 which helped create a nice table and I was able to select each item from the summary statistics function. Overall, this section was great to work on with a clear build up in difficulty.\n"
+ "These tasks set a good basis for the assignment with the previous worksheets helping with the understanding of lists and the different operators that could be used to solve the summary statistics. I placed FR1.8 at the top of the assignment as it was easier to use the data given then make up my own, opening this file was relatively easy with the main issues occurring when taking the list from strings to integers to enable me to perform different calculations on it. The first few questions were straightforward with basic mathematics which is a strength of mine, the difficulty was more around writing the code but with different links and worksheets it became clearer, and the use of the round operator made the values more pleasing to look at. Once again throughout these questions there was a lot of use of the basic operators that Python offers which made the code more concise. When completing the questions for the percentiles I struggled with the indexes as I was uncertain if the value produced included the 0 index or if it did not. I decided to take 1 off these values produce to get it into index form as I felt this was the correct way to do this and then merely found this value in the sorted list. There was a great link on the internet that gave me a good idea how to create a nicely formatted table for FR1.7 which involved calling the answers from the previous tasks and meant that this table would work for new .dat files.\n",
+ "\n",
+ "The tasks that involved task1.csv file it was challenging, with Fr1.10 causing issues and so I became creative in storing this csv to memory. I felt the best way to approach this task was to remove the header, create a loop which looked at each row, removing the comma and new line character. Then I created a loop for each value in the row and appended this to a nested list. Next, I created a dictionary with the headings to store each value. For the summary statistics I created a function that had empty dictionaries which would be filled with the summary statistics for each header. I was unable to get standard deviation working within these, as I couldn’t figure out how to create the sum((v-mean[h])^2) to work for each heading however I have left the code in but commented out. I wish I had made the function print clearer results however this task did cause me to think outside the box and create a more complex function. I followed some of the same links as for different tasks which helped in creating nice tables and showed me how to select each item from the summary statistics function. Overall, this section was great to work on with a clear build up in difficulty and it has become clear the areas where I understood and the ones that required more work with more time available.\n",
+ "\n"
]
},
{
diff --git a/.ipynb_checkpoints/UFCFVQ-15-M Programming Task 2 Template (1)-checkpoint.ipynb b/.ipynb_checkpoints/UFCFVQ-15-M Programming Task 2 Template (1)-checkpoint.ipynb
index c6c9dc9b89687c0cc3b8afff040fbe87adf9dae3..458f4de113eeb6ee6b535571a3ffd04329d4bf09 100644
--- a/.ipynb_checkpoints/UFCFVQ-15-M Programming Task 2 Template (1)-checkpoint.ipynb
+++ b/.ipynb_checkpoints/UFCFVQ-15-M Programming Task 2 Template (1)-checkpoint.ipynb
@@ -1825,11 +1825,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "### FR2.1 - FR2.8\n",
- "Overall, this part of the assignment was a lot easier than the previous one as the use of modules meant the code became simpler and it was easier to make the code clear and concise. The first couple of tasks were very straightforward and there was a lot of help available online when using Pandas. FR2.3 was slightly more difficult to complete as trying to ensure that the correct merge was done as it could affect how the two data frames came together. With regards to FR2.4-2.5 I had previously used this in another assignment so felt very confident in my ability to complete this and for FR2.6 there was a very helpful link that demonstrated how to change column names. Furthermore FR2.7 was easier to understand with little tweaks occurring to ensure the question was answered and further learning how the axis affected the results. FR2.8 involved saving the data frame to a new csv file and removing the index from the csv file to stop unwanted columns forming if more edits were to happen to the csv file.\n",
+ "Overall, this part of the assignment was a lot easier than the previous one as the use of modules meant the code became simpler and it was easier to make the code clear and concise. The first couple of tasks were very straightforward and there was a lot of help available online when using Pandas. I felt Pandas was a great module with help available for almost everything I could think of and since using it in a previous assignment my confidence in the first half of the assignment meant it was easy to work through it without much difficulty. One of the things I learned when using pandas is when you save a new file it also saves the indexes so there was a conscious effort to stop this as it clutters the data and is not required. The skills learned and used within this assignment is invaluable and demonstrates how important Pandas is within Data Science\n",
"\n",
- "### FR2.9 - FR2.12\n",
- "For FR2.9 there was a great website that helped a lot with answering this question as I was unsure how to select specific columns to be involved in the groupby command. FR2.10 was a lot of trial and error to produce a clear visualisation that showed the results from the previous task. Overall, I felt the best visualisation would be with a dual axis with two bar charts that shared the same x-axis, I felt like this helped show the changes for each age group for both the variables and I tried to ensure the plot had a title and renamed the headers to make it look clearer. When investigating the effects of the engagement on attainment I felt it was best to plot the two variables in a scatter plot with a regression line to visualise the data and see if there was a visible relationship between them. The regression line suggested that there could be correlation between click event and final mark and so the next stage was to get correlation coefficient to see the relationship between the two variables, numpy was a great module to find the correlation coefficients. To test the hypothesis, I performed a linear regression model but it was difficult to figure out how to perform this as I was unsure if scipy was the numpy stats module but the majority of websites said it was. To further investigate the data, I checked the R-squared value which suggested the model was not a good fit but this can be due to the very large dataset and further that the previous task suggested that there wasn’t a strong relationship between these two variables. In the future for FR2.12 I would include more statistical tests and look more at the goodness-of-fit to ensure the most appropriate model is being used, while also looking at residual plots from the regression model.\n"
+ "For FR2.9 there was a great website that helped a lot with answering this question as I was unsure how to select specific columns to be involved in the groupby command. When producing the visualisation there was a lot of trial and error to produce a clear visualisation that showed the results in a well formatted method. Overall, I felt the best visualisation when presenting the age group investigation would be with a dual axis with two bar charts that shared the same x-axis. I felt like this helped show the changes for each age group for both the variables and I tried to ensure the plot had a title and renamed the headers to make it look clearer. When investigating the effects of the engagement on attainment I felt it was best to plot the two variables in a scatter plot with a regression line to visualise the data and see if there was a visible relationship between them. With regards to the final two tasks, I was confident in the statistical part of the tasks but struggled with the programming side of this as using Seaborn and matplotlib was a new and more complex area and more online research was required to start this. To test the hypothesis, I performed a linear regression model but it was difficult to figure out how to perform this as I was unsure if scipy was the numpy stats module but the majority of websites said it was. Scipy helped make the final two tasks easier to work on as it did the work for you, I felt it helped to produce the values I wanted whilst also keeping the code concise. In the future for FR2.12 I would include more statistical tests and look more at the goodness-of-fit to ensure the most appropriate model is being used, while also looking at residual plots from the regression model. This would improve the validity of the results but would also require more complex code which would involves more research and confidence within this area."
]
},
{
diff --git a/UFCFVQ-15-M Programming Task 1 Template.ipynb b/UFCFVQ-15-M Programming Task 1 Template.ipynb
index 01f60e096d4ddb0f3a0e9abaf543f18874984944..f797f496952d066c2997368bf0b04f8ce8277b63 100644
--- a/UFCFVQ-15-M Programming Task 1 Template.ipynb
+++ b/UFCFVQ-15-M Programming Task 1 Template.ipynb
@@ -307,7 +307,17 @@
"any new line characters using .strip(). Next each item within all the rows using the index will be inserted into the the sublists of the empty data list.\n",
"Next I will create a dictionary to store all this data in and assign each sublist with the correct headings.\n",
"As good practive the CSV file should then be closed. Finally you will use the return command which will exit the function and allow the function to produce values based on the return key used.\n",
- "\n"
+ "\n",
+ "file_csv = \"usefile.csv\"\n",
+ "def file(file_csv):\n",
+ " file_open = open(file_csv, \"r\")\n",
+ " file_open.readline()\n",
+ " data = [[],[],[],[],[],[],[],[],[]]\n",
+ " for row in file_open:\n",
+ " items = row.strip().split(\",\")\n",
+ " items = [float(i) for i in items]\n",
+ " file_open.close()\n",
+ " return task"
]
},
{
@@ -505,10 +515,10 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "### FR1.1 - FR1.8\n",
- "These tasks set a good basis for the assignment while the previous worksheets helped with the understanding of lists and the different operators that could be used to solve the summary statistics. I placed FR1.8 at the top of the assignment as it was easier to use the data given then make up my own, opening this file was relatively easy with the main issues occurring when taking the list from strings to integers to enable me to perform different calculations on it. For the standard deviation the use of a loop made the code more concise and enabled me to create a variable and use the += assignment which adds a number to this variable until the loop gone through each item in the list. Furthermore, I also used the round operator which helped make the value easier to read which meant the table further below will looked pleasing. Once again throughout these questions there was a lot of use of the basic operators that Python offers which made the code more concise. When completing the questions for the percentiles I struggled with the indexes as I was uncertain if the value produced included the 0 index or if it did not. I decided to take 1 off these values produce to get it into index form as I felt this was the correct way to do this and then merely found this value in the sorted list. There was a great link on the internet that gave me a good idea how to create a nicely formatted table for FR1.7 which involved calling the answers from the previous tasks and meant that this table would work for new .dat files.\n",
- "### FR1.9 - FR1.12\n",
- "The tasks that involved task1.csv file it was challenging, with Fr1.10 causing issues and so I became creative in storing this csv to memory. I felt the best way to approach this task was to remove the header,create a loop which looked at each row, removing the comma and new line character. Then I created a loop for each value in the row and appended this to a nested list. Next, I created a dictionary with the headings to store each value. For the summary statistics I created a function that had empty dictionaries which would be filled with the summary statistics for each header. I was unable to get standard deviation working within these, as I couldn’t figure out how to create the sum((v-mean[h])^2) to work for each heading however I have left the code in but commented out. I wish I had made the function print clearer results however this task did cause me to think outside the box and create a more complex function. For FR1.12 I followed the same link as for FR1.7 which helped create a nice table and I was able to select each item from the summary statistics function. Overall, this section was great to work on with a clear build up in difficulty.\n"
+ "These tasks set a good basis for the assignment with the previous worksheets helping with the understanding of lists and the different operators that could be used to solve the summary statistics. I placed FR1.8 at the top of the assignment as it was easier to use the data given then make up my own, opening this file was relatively easy with the main issues occurring when taking the list from strings to integers to enable me to perform different calculations on it. The first few questions were straightforward with basic mathematics which is a strength of mine, the difficulty was more around writing the code but with different links and worksheets it became clearer, and the use of the round operator made the values more pleasing to look at. Once again throughout these questions there was a lot of use of the basic operators that Python offers which made the code more concise. When completing the questions for the percentiles I struggled with the indexes as I was uncertain if the value produced included the 0 index or if it did not. I decided to take 1 off these values produce to get it into index form as I felt this was the correct way to do this and then merely found this value in the sorted list. There was a great link on the internet that gave me a good idea how to create a nicely formatted table for FR1.7 which involved calling the answers from the previous tasks and meant that this table would work for new .dat files.\n",
+ "\n",
+ "The tasks that involved task1.csv file it was challenging, with Fr1.10 causing issues and so I became creative in storing this csv to memory. I felt the best way to approach this task was to remove the header, create a loop which looked at each row, removing the comma and new line character. Then I created a loop for each value in the row and appended this to a nested list. Next, I created a dictionary with the headings to store each value. For the summary statistics I created a function that had empty dictionaries which would be filled with the summary statistics for each header. I was unable to get standard deviation working within these, as I couldn’t figure out how to create the sum((v-mean[h])^2) to work for each heading however I have left the code in but commented out. I wish I had made the function print clearer results however this task did cause me to think outside the box and create a more complex function. I followed some of the same links as for different tasks which helped in creating nice tables and showed me how to select each item from the summary statistics function. Overall, this section was great to work on with a clear build up in difficulty and it has become clear the areas where I understood and the ones that required more work with more time available.\n",
+ "\n"
]
},
{
diff --git a/UFCFVQ-15-M Programming Task 2 Template (1).ipynb b/UFCFVQ-15-M Programming Task 2 Template (1).ipynb
index c6c9dc9b89687c0cc3b8afff040fbe87adf9dae3..458f4de113eeb6ee6b535571a3ffd04329d4bf09 100644
--- a/UFCFVQ-15-M Programming Task 2 Template (1).ipynb
+++ b/UFCFVQ-15-M Programming Task 2 Template (1).ipynb
@@ -1825,11 +1825,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "### FR2.1 - FR2.8\n",
- "Overall, this part of the assignment was a lot easier than the previous one as the use of modules meant the code became simpler and it was easier to make the code clear and concise. The first couple of tasks were very straightforward and there was a lot of help available online when using Pandas. FR2.3 was slightly more difficult to complete as trying to ensure that the correct merge was done as it could affect how the two data frames came together. With regards to FR2.4-2.5 I had previously used this in another assignment so felt very confident in my ability to complete this and for FR2.6 there was a very helpful link that demonstrated how to change column names. Furthermore FR2.7 was easier to understand with little tweaks occurring to ensure the question was answered and further learning how the axis affected the results. FR2.8 involved saving the data frame to a new csv file and removing the index from the csv file to stop unwanted columns forming if more edits were to happen to the csv file.\n",
+ "Overall, this part of the assignment was a lot easier than the previous one as the use of modules meant the code became simpler and it was easier to make the code clear and concise. The first couple of tasks were very straightforward and there was a lot of help available online when using Pandas. I felt Pandas was a great module with help available for almost everything I could think of and since using it in a previous assignment my confidence in the first half of the assignment meant it was easy to work through it without much difficulty. One of the things I learned when using pandas is when you save a new file it also saves the indexes so there was a conscious effort to stop this as it clutters the data and is not required. The skills learned and used within this assignment is invaluable and demonstrates how important Pandas is within Data Science\n",
"\n",
- "### FR2.9 - FR2.12\n",
- "For FR2.9 there was a great website that helped a lot with answering this question as I was unsure how to select specific columns to be involved in the groupby command. FR2.10 was a lot of trial and error to produce a clear visualisation that showed the results from the previous task. Overall, I felt the best visualisation would be with a dual axis with two bar charts that shared the same x-axis, I felt like this helped show the changes for each age group for both the variables and I tried to ensure the plot had a title and renamed the headers to make it look clearer. When investigating the effects of the engagement on attainment I felt it was best to plot the two variables in a scatter plot with a regression line to visualise the data and see if there was a visible relationship between them. The regression line suggested that there could be correlation between click event and final mark and so the next stage was to get correlation coefficient to see the relationship between the two variables, numpy was a great module to find the correlation coefficients. To test the hypothesis, I performed a linear regression model but it was difficult to figure out how to perform this as I was unsure if scipy was the numpy stats module but the majority of websites said it was. To further investigate the data, I checked the R-squared value which suggested the model was not a good fit but this can be due to the very large dataset and further that the previous task suggested that there wasn’t a strong relationship between these two variables. In the future for FR2.12 I would include more statistical tests and look more at the goodness-of-fit to ensure the most appropriate model is being used, while also looking at residual plots from the regression model.\n"
+ "For FR2.9 there was a great website that helped a lot with answering this question as I was unsure how to select specific columns to be involved in the groupby command. When producing the visualisation there was a lot of trial and error to produce a clear visualisation that showed the results in a well formatted method. Overall, I felt the best visualisation when presenting the age group investigation would be with a dual axis with two bar charts that shared the same x-axis. I felt like this helped show the changes for each age group for both the variables and I tried to ensure the plot had a title and renamed the headers to make it look clearer. When investigating the effects of the engagement on attainment I felt it was best to plot the two variables in a scatter plot with a regression line to visualise the data and see if there was a visible relationship between them. With regards to the final two tasks, I was confident in the statistical part of the tasks but struggled with the programming side of this as using Seaborn and matplotlib was a new and more complex area and more online research was required to start this. To test the hypothesis, I performed a linear regression model but it was difficult to figure out how to perform this as I was unsure if scipy was the numpy stats module but the majority of websites said it was. Scipy helped make the final two tasks easier to work on as it did the work for you, I felt it helped to produce the values I wanted whilst also keeping the code concise. In the future for FR2.12 I would include more statistical tests and look more at the goodness-of-fit to ensure the most appropriate model is being used, while also looking at residual plots from the regression model. This would improve the validity of the results but would also require more complex code which would involves more research and confidence within this area."
]
},
{