diff --git a/UFCFVQ-15-M_Programming_Task_1.ipynb b/UFCFVQ-15-M_Programming_Task_1.ipynb
index c3869615573f8ffded72577abdf2182c96b8aba2..bf8261028bb7b6dc2d30c5edd625a6efc314b1c0 100644
--- a/UFCFVQ-15-M_Programming_Task_1.ipynb
+++ b/UFCFVQ-15-M_Programming_Task_1.ipynb
@@ -977,7 +977,224 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "add markdown text here"
+ "Draft version below - 966 words (including everything)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "•\tYou are expected to identify the strengths/weaknesses of your approach. For this programming task, you are expected to write a reflective report which focuses on the process taken to develop a solution to the task. Please reflect on your experiences rather than simply describing what you did. The report should: \n",
+ "o\tinclude an explanation of how you approached the task.\n",
+ "o\tidentify any strengths/weaknesses of the approach used.\n",
+ "o\tconsider how the approach used could be improved.\n",
+ "o\tsuggest alternative approaches that could have been taken instead of the one you used.\n",
+ "\n",
+ "o\tup to 8 marks for the Development Process Report\n",
+ "\tMarks will be awarded for appropriate use of technical language, critical reflection on development process and quality of engagement with the reflective process\n",
+ "\n",
+ "word count - 500 max\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "FR1: arithmetic mean\n",
+ "\n",
+ "* easy to implement - good confidence boost\n",
+ "* allows for getting into a flow of coding, including comments, docstrings, naming conventions (variables and functions), testing, etc.\n",
+ "* I kept returning to the naming and commenting of early requirements - as they developed. \n",
+ "* Challenge was getting the balance right - on the one hand, I wanted to ensure that I was including adequate comments but on the other hand, I did not want to over document. Bearing in mind that this is an assessed element of coursework submitted for a master's module, I think I have the right balance. In 'the real world,' I would probably shorten some of the docstrings and remove some in line comments. \n",
+ "* My tactic has been to try to name variables and functions in an unambiguous, meaningful manner so that documentation is unnecessary.\n",
+ "* I haven't settled on a style yet - e.g. casing (proper, camel, etc.) and use of underscores. \n",
+ "* When writing functions, I wanted them to have reuse value - that is, to keep them general. With some requirements, this was not possible - some of my ideas would have deviated from the brief. Examples later. \n",
+ "* I wanted to use helpful python code where relevant, e.g. try/except blocks with specific (anticipated) errors captured. \n",
+ "\n",
+ "* WHat I would do differently:\n",
+ " * I need to be mindful of using restricted/reserved words as variable names. I have used 'list' in FR1 function. This is not a problem in this instance but it could be in others. I will need to be more careful in future.\n",
+ " * I also would like to assess the efficiency of my functions - there are so many ways to write functions producing the same outcomes in python. Absolutely amazed at how flexible and powerful it is - but at the moment, I am pleased when I can get my code to work. Efficiency, elegance, etc. will come with time and experience - but I could start using time to assess processing times for different approaches. \n",
+ "\n",
+ "FR2: read single column from CSV file\n",
+ "\n",
+ "* Again, I wanted to make a reusable function, so I added 'delimiter' as an optional parameter. I prefer to use pipes (|) as delimiters as they are very seldom used in text. This is something I have found handy in data migration work I have undertaken in my current job. Quite often - common characters like commas and 'invisible' spacing characters like tabs, new line, carriage returns, etc. can wreak havoc when migrating data. \n",
+ "* I liked this function as it was a good opportunity to use the 'with' statement. I like that it 'closes' the file when finished, without a need for an explicit close statement. \n",
+ "* The challenge for me in this requirement was working out how to extract the header row and then proceed to the data rows. \n",
+ "* It was a good example of where I knew what I wanted to do and it seemed simple enough, but took some trial and error before I got there (plenty more to come!)\n",
+ "* Also added a specific FileNotFoundError exception as I expect this to be the most common error.\n",
+ "* I also added a if/else statement to account for whether the file has a header - returning results accordingly. While this is not in the brief, I was thinking about a future proofed function which can handle both varieties.\n",
+ "\n",
+ "FR3: read CSV data from a file into memory\n",
+ "\n",
+ "* Much of the above applies - the function docstring is probably a bit long but I wanted to be clear and complete in my documentation.\n",
+ "* This function builds on FR2 - but returns a dictionary by iterating through the data rows of the file. \n",
+ "* I worked through many versions of this - for loops, enumerate, etc. - but settled for a list comprehension, once I get it working the way it should. I am not sure if it is the most efficient way to do it, but it works. However, I like the conciseness of list comprehensions - although they may not be as readable as a for loop.\n",
+ "* In terms of variable names, I am not always clear on the standards - i, j, row, line, etc. and I vary between them - which annoys me. Consistency, clarity, transparency, etc. are very important to me. I am not so happy with 'variable_data_dict' as a name, but couldn't find anything better - 'my_dict', 'data_dict', etc. - I am critical of them all. \n",
+ "\n",
+ "\n",
+ "* one problem I had with FR3 was discovered when using the function in various of FR4. The original dictionary produced string key-value pairs - which when printing looked fine to me. It took me some time to realise that the values were string - that they were enclosed in single quotation marks. This was certainly a source of frustration - when something simply was not behaving the way I was anticipating.\n",
+ " * However, once I noticed, I tried to amend the list comprehension by convernting to integers with int() - which caused an error - \"invalid literal for int() with base 10: '60.5'\"\n",
+ " * Therefore I converted the data to floats - which may not be quite right for ages, but will not impact the calculations.\n",
+ "* \n",
+ "\n",
+ "FR4: Pearson Correlation Coefficient\n",
+ "\n",
+ "* I spent a lot of time on this function and tried to make it work using several methodologies. \n",
+ "* Pseudo code was particularly helpful in this instance, but I have a tendency of starting with pseudo code and then deviating into a confused space of trial and error rather than updating the pseudo code. Definitely something I can work on and take away into my current day job. \n",
+ "* Firstly I needed to remind myself how to calculate Pearson's r having only have had it calculated for me in SPSS or R. There a couple of approaches - but the strategy is to break down the formula into standalone calculations and build the function to produce these before putting it all together to return R. \n",
+ "* I spent a lot of time trying different approaches. Again, like last time I started with for loops and enumerate() but found that I couldn't always achieve my intentions - that I would get slightly lost in the hierarchy and multilined approach, so I used list comprehensions instead and in the end, I am pleased with the result.\n",
+ "\n",
+ "* As previously, I tried thinking about generalising and future-proofing the function for future use and that includes considering what tests / checks / validation I could put in place. The most obvious source of error will be the csv file so I added some assertions to ensure that the csv file has the correct data before proceeding to calculate Pearson's r. \n",
+ "* I tested my version of Pearsons with numpy corrcoef() and it produces the same results for a small data set, which gives you that excitement of having achieved your intentions.\n",
+ "\n",
+ "\n",
+ "FR5: Pearson Correlation Coefficients for a file\n",
+ "\n",
+ "* This function builds on FR4 by calculating Pearson's r for each pair of columns in a csv file.\n",
+ "* I enjoyed using FR3 and FR4 (which in turn use FR1) within this function. That is pretty cool.\n",
+ "* It does not have many lines of code but it took me a while to get it working. I had to think about how to iterate through the columns of the csv in terms of what has been returned by FR3 and FR4 and then getting the column names into a list of tuples.\n",
+ "* It was not plain sailing and there was a fair bit of frustration - another good example of where I knew exactly what I wanted to do but it was hard to get there. I had a few versions on the go - I think I could have benefited by sticking with one and updating the pseudo code. \n",
+ "* As previously, I moved from for loops to list comprehensions as I had better success getting the results I wanted, especially returning lists of tuples. Initially, I was returning individual lists instead of lists of tuples. \n",
+ "* The joy of success after some struggle can be very satisfying. \n",
+ "\n",
+ "I would probably look to round some of the output in future iterations.\n",
+ "Actually - I just added the rounding now!\n",
+ "\n",
+ "And finally\n",
+ "\n",
+ "FR6: printing the results\n",
+ "\n",
+ "* I found this FR very challenging and a bit frustrating.\n",
+ "* I could have benefitted significantly by sticking with my pseudo code and keeping it up to date - but once I got stuck into a cycle of trial and error, minor success and subsequent set-back, I did not rever to pseudocode and focused on cracking the issue which was hampering me. \n",
+ "* This function was created through trial and error - entirely\n",
+ "* I broke it down into separate sections and tried to think how I would iterate through the list of tuples to get a decent end result.\n",
+ "* Some sections worked, others didn't, changes then impacted the rest of the function and I was back to square one - hardforking!\n",
+ "* Towards the end, I had functioning portions - the top table header, the column headers, the rows headers, the r-values. This was a mess of for loops, list comprehensions, and string formatting which I needed to work through step by step.\n",
+ "* I found cells in Jupyter notebooks to be invaluable - the ability to run portions of code and easily move it around is very effective and powerul. I also found the ability to comment out sections of code very useful - I could easily comment out sections of code and then uncomment them to see how they impacted the rest of the function.\n",
+ "* Finally, the pieces (rows of code) were lining up - at this point, I resorted to tweaking and assessing the impact of each change. I discovered that I needed data specific to the input table - rather than fixed widths - so created a separate function to calculate max column widths and then played with that to get the table to look passable. \n",
+ "\n",
+ "Whew - got there in the end! \n",
+ "\n",
+ "Although this was more challenging to get motivated over as I am sure I would use some library to print tables without the effort and challenge - i.e. pandas, prettytable, etc. However, in hindsight I am glad I persevered as I have learned a lot about string formatting and how to iterate through lists of tuples. It allowed me to trouble-shoot, iterate through a requirement, reassess and rework through many cycles. I am sure I will use this knowledge in the future.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "add markdown text here\n",
+ "\n",
+ "coding - use of x, y, common practices, etc.\n",
+ "naming conventions (camel, underscore, etc.)\n",
+ "level/amount of commenting\n",
+ "efficiency of code\n",
+ "\n",
+ "\n",
+ "Thought I was attentive to detail\n",
+ "Eg print statements \n",
+ "\n",
+ "Challenging to get the iteration done\n",
+ "Know what I want to do but achieving it programmatically is hard and slow\n",
+ "\n",
+ "Balancing the need to know vs how to find out what I need to know\n",
+ "For example researching code, assessing, evaluating, understanding, not assuming. \n",
+ "\n",
+ "Knowing fundamentals but not everything. Realistically would use libraries where they do what you need them to. Maybe not most efficient, but most practical or ease of use or pretty. Maybe not quickest run time but quicker to get up and running. There's a balance to be struck - knowing what your requirements are and adapting to them...metaphors I use - chefs and recipes and ingredients and cake shops. Sometimes you need to buy the cake pre-made and decorated, sometimes you make it from scratch, sometimes you buy it and decorate it, sometimes you farm the eggs. \n",
+ "\n",
+ "Or building a shed: Building shed analogy \n",
+ "\n",
+ "Don't need to know how to build the pieces individually\n",
+ "Carpentry power tools skills\n",
+ "May get a better product\n",
+ "May get exactly what I want\n",
+ "But requires a sig investment and aptitude whereas can by prefab, deconstructed where hard work done. You can focus on the outcome and the nuance and styling\n",
+ "\n",
+ "Or decorating a room - v being interior designer\n",
+ "\n",
+ "Coding and documenting. Importance of. Dry wet. Balance. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### <b> Introduction </b>\n",
+ "\n",
+ "The purpose of this report is to provide a short, critical self-assessment of all aspects of the code development process undertaken for Task 1 of the assessed coursework for `UFCFVQ-15-M Programming for Data Science`. The report will include a brief description of the code, a reflection on the development process and an evaluation of the code, including strengths and weaknesses and alternatives to consider. \n",
+ "\n",
+ "\n",
+ "### <b> Code Description </b>\n",
+ "\n",
+ "The requirement for `Task_1` is to calculate Pearson Correlation Coefficients (PCCs) for all pairs of variables in a given data file, without using any external libraries such as Pandas, and print a decent-looking, customisable table of the results. \n",
+ "\n",
+ "The code was written in an interactive Jupyter notebook using a Python 3.11 kernel. \n",
+ "\n",
+ "The coursework is structured so that the code must be developed in a series of steps, each building on or incorporating its predecessor. There are six functional requirements (FRs) as follows:\n",
+ "\n",
+ "| FR | Description |\n",
+ "|-----|-----------------------|\n",
+ "| FR1 | Arithmetic mean | \n",
+ "| FR2 | Read column from file | \n",
+ "| FR3 | Read file | \n",
+ "| FR4 | PCC for two lists | \n",
+ "| FR5 | PCC for file | \n",
+ "| FR6 | Print table | \n",
+ "\n",
+ "\n",
+ "Pearson's correlation coefficient measures linear association between two variables, ranging from -1 to 1, where -1 is perfect negative linear correlation and 1 is perfect positive linear correlation. The formula for calculating the PCC is as follows:\n",
+ "\n",
+ "Given paired data, consisting of $n$ pairs: \n",
+ "$$\n",
+ "\n",
+ "r =\n",
+ " \\frac{ \\sum_{i=1}^{n}(x_i-\\bar{x})(y_i-\\bar{y}) }{%\n",
+ " \\sqrt{\\sum_{i=1}^{n}(x_i-\\bar{x})^2}\\sqrt{\\sum_{i=1}^{n}(y_i-\\bar{y})^2}}\n",
+ "$$\n",
+ "\n",
+ "\n",
+ "source: [Wikipedia](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient#Definition)\n",
+ "\n",
+ "\n",
+ " \n",
+ "### <b> Reflection on Development Process</b>\n",
+ "\n",
+ "My development process made use of the inherent stratified nature of the task, allowing me to plan, develop and test each FR independently in separate code cells, before combining where appropriate. I found ths particularly useful for the more complex FRs - 4, 5, and 6 which required significant iteration, investigation and testing before achieving the desired results. This also worked very well in terms of source and version control, complementing the use of Git.\n",
+ "\n",
+ "My strategy was to use a localised crisp-dm approach which included initial articulation of requirements followed by iterative pseudocode, Python code and testing cycles until the desired results were achieved. I find this approach very effective and it is one I use in my current employment. However, having said that, I find that I can occasionally spiral out of control in the testing phase, which can be time-consuming, frustrating and ultimately less productive. I will continue to try to be mindful of this. \n",
+ "\n",
+ "\n",
+ "[create and insert a crisp-dm diagram here]\n",
+ "\n",
+ "I tried to be disciplined in my approach and for the most part feel like I was successful. I made deliberate attempts to make use of newly acquired tools and techniques like Git, VS Code, Jupyter notebooks, Markdown - ensuring that I commented on my code, made use of Markdown cells during the development phase, and committing changes to Git in appropriate intervals.\n",
+ "\n",
+ "\n",
+ "### <b> Code Evaluation </b>\n",
+ "\n",
+ "When it comes to code evaluation, I think it is important to acknowledge that the Task_1 code has been developed to meet the coursework specification. As such, it has been produced with certain limitations and constraints and does not necessarily reflect my approach in a different context, such as my employment or personal projects. However, it is important to reflect on what has been produced. \n",
+ "\n",
+ "Overall, I am pleased with the code that I have produced. It achieves the requirements (as I have interpreted them) and it <i>feels</i> fairly efficient and robust. \n",
+ "\n",
+ "I tried to be mindful of certain principles when undertaking this work:\n",
+ "\n",
+ "* Future-proofing the functions, i.e. making them flexible and adaptable to different data types and formats, perhaps accepting different parameters, etc. \n",
+ "* Thinking about likely pitfalls from the users perspective and pre-empting likely errors with assertions and exception handling\n",
+ "* Unambiguous, self-explanatory naming of functions and variables, where possible\n",
+ "* Balanced comments and docstrings - considering approaches like DRY (Dont Repeat Yourself), WET (Write Everything Twice), KISS (Keep it Simple, Stupid) bearing in mind possible readers of the code\n",
+ "\n",
+ "I think I have been reasonably successful in achieving these goals. I have tried to make the code as flexible as possible, accepting different parameters and data types, and I have tried to make the code as robust as possible, using assertions and exception handling to pre-empt likely errors. I have saved alternate versions of the functions for future use and reference, which take additional parameters or employ a different approach. \n",
+ "\n",
+ "In terms of naming convention, documentation and comments, I think I struck the right balance. If anything, I have probably over-commented slightly but this is because I am erring on the side of caution. I have yet to settle on a naming convention and standard and find that I flipflop between different methods like camelCase, snake_case, PascalCase, etc. I am also aware that there seem to be conventions for variables and functions, especially iterables, which I am not fully using. \n",
+ "\n",
+ "As for the code itself, I think it is fairly efficient and robust. Of course, there is always room for improvement and I would very much like to know of any suggestions or improvements, or better approaches.\n",
+ "\n",
+ "\n",
+ "#### <b> Summary </b>\n",
+ "\n",
+ "Whilst I have a background in elements of data and systems, I am only getting started in the programming part of this journey. I am super keen to learn for my personal and professional development and want to pick up best practices, adopt standard approaches and avoid pitfalls and repeating mistakes. When it comes to Python, I am amazed at how many different ways there are to solve a particular scenario. This can make it impossible to identify a 'single best approach' - something I will need to get used to and embrace. \n",
+ "\n",
+ "\n"
]
},
{
diff --git a/UFCFVQ-15-M_Programming_Task_1_submit.ipynb b/UFCFVQ-15-M_Programming_Task_1_submit.ipynb
index c85264fc076355a4eeb78eda1c66fda1eb5c81d3..faf5866758c549f34f994c0e63c3b7e1934bedcf 100644
--- a/UFCFVQ-15-M_Programming_Task_1_submit.ipynb
+++ b/UFCFVQ-15-M_Programming_Task_1_submit.ipynb
@@ -542,17 +542,26 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "•\tYou are expected to identify the strengths/weaknesses of your approach. For this programming task, you are expected to write a reflective report which focuses on the process taken to develop a solution to the task. Please reflect on your experiences rather than simply describing what you did. The report should: \n",
- "o\tinclude an explanation of how you approached the task.\n",
- "o\tidentify any strengths/weaknesses of the approach used.\n",
- "o\tconsider how the approach used could be improved.\n",
- "o\tsuggest alternative approaches that could have been taken instead of the one you used.\n",
+ "Plan / Outline: \n",
"\n",
- "o\tup to 8 marks for the Development Process Report\n",
- "\tMarks will be awarded for appropriate use of technical language, critical reflection on development process and quality of engagement with the reflective process\n",
+ "`Introduction:` overview of the purpose and objectives of the code, as well as the context in which it was developed (e.g., as part of assessed coursework).\n",
"\n",
- "word count - 500 max\n",
+ "`Description of the code:` detailed description of the code, including its main features, functionality, and any relevant technical details (e.g., libraries used).\n",
"\n",
+ "`Reflection on the development process:` reflect on experience of developing the code, including any challenges and overcoming; discuss any lessons learned or insights gained during the development process.\n",
+ "\n",
+ "`Evaluation of the code:` evaluate the code based on its functionality, performance, and overall quality. discuss limitations or areas for improvement.\n",
+ "\n",
+ "`Conclusion:` summarise the key points of reflection, provide any final thoughts or recommendations for future work.\n",
+ "\n",
+ "\n",
+ "Version 1 below - 966 words (total with everything)\n",
+ "\n",
+ "TODO\n",
+ "\n",
+ "Review version 1\n",
+ "Review notes (in non-submit version)\n",
+ "Reduce in word count\n",
"\n"
]
},
@@ -560,118 +569,81 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "FR1: arithmetic mean\n",
+ "### <b> Introduction </b>\n",
"\n",
- "* easy to implement - good confidence boost\n",
- "* allows for getting into a flow of coding, including comments, docstrings, naming conventions (variables and functions), testing, etc.\n",
- "* I kept returning to the naming and commenting of early requirements - as they developed. \n",
- "* Challenge was getting the balance right - on the one hand, I wanted to ensure that I was including adequate comments but on the other hand, I did not want to over document. Bearing in mind that this is an assessed element of coursework submitted for a master's module, I think I have the right balance. In 'the real world,' I would probably shorten some of the docstrings and remove some in line comments. \n",
- "* My tactic has been to try to name variables and functions in an unambiguous, meaningful manner so that documentation is unnecessary.\n",
- "* I haven't settled on a style yet - e.g. casing (proper, camel, etc.) and use of underscores. \n",
- "* When writing functions, I wanted them to have reuse value - that is, to keep them general. With some requirements, this was not possible - some of my ideas would have deviated from the brief. Examples later. \n",
- "* I wanted to use helpful python code where relevant, e.g. try/except blocks with specific (anticipated) errors captured. \n",
+ "The purpose of this report is to provide a short, critical self-assessment of all aspects of the code development process undertaken for Task 1 of the assessed coursework for `UFCFVQ-15-M Programming for Data Science`. The report will include a brief description of the code, a reflection on the development process and an evaluation of the code, including strengths and weaknesses and alternatives to consider. \n",
"\n",
- "* WHat I would do differently:\n",
- " * I need to be mindful of using restricted/reserved words as variable names. I have used 'list' in FR1 function. This is not a problem in this instance but it could be in others. I will need to be more careful in future.\n",
- " * I also would like to assess the efficiency of my functions - there are so many ways to write functions producing the same outcomes in python. Absolutely amazed at how flexible and powerful it is - but at the moment, I am pleased when I can get my code to work. Efficiency, elegance, etc. will come with time and experience - but I could start using time to assess processing times for different approaches. \n",
"\n",
- "FR2: read single column from CSV file\n",
+ "### <b> Code Description </b>\n",
"\n",
- "* Again, I wanted to make a reusable function, so I added 'delimiter' as an optional parameter. I prefer to use pipes (|) as delimiters as they are very seldom used in text. This is something I have found handy in data migration work I have undertaken in my current job. Quite often - common characters like commas and 'invisible' spacing characters like tabs, new line, carriage returns, etc. can wreak havoc when migrating data. \n",
- "* I liked this function as it was a good opportunity to use the 'with' statement. I like that it 'closes' the file when finished, without a need for an explicit close statement. \n",
- "* The challenge for me in this requirement was working out how to extract the header row and then proceed to the data rows. \n",
- "* It was a good example of where I knew what I wanted to do and it seemed simple enough, but took some trial and error before I got there (plenty more to come!)\n",
- "* Also added a specific FileNotFoundError exception as I expect this to be the most common error.\n",
- "* I also added a if/else statement to account for whether the file has a header - returning results accordingly. While this is not in the brief, I was thinking about a future proofed function which can handle both varieties.\n",
+ "The requirement for `Task_1` is to calculate Pearson Correlation Coefficients (PCCs) for all pairs of variables in a given data file, without using any external libraries such as Pandas, and print a decent-looking, customisable table of the results. \n",
"\n",
- "FR3: read CSV data from a file into memory\n",
+ "The code was written in an interactive Jupyter notebook using a Python 3.11 kernel. \n",
"\n",
- "* Much of the above applies - the function docstring is probably a bit long but I wanted to be clear and complete in my documentation.\n",
- "* This function builds on FR2 - but returns a dictionary by iterating through the data rows of the file. \n",
- "* I worked through many versions of this - for loops, enumerate, etc. - but settled for a list comprehension, once I get it working the way it should. I am not sure if it is the most efficient way to do it, but it works. However, I like the conciseness of list comprehensions - although they may not be as readable as a for loop.\n",
- "* In terms of variable names, I am not always clear on the standards - i, j, row, line, etc. and I vary between them - which annoys me. Consistency, clarity, transparency, etc. are very important to me. I am not so happy with 'variable_data_dict' as a name, but couldn't find anything better - 'my_dict', 'data_dict', etc. - I am critical of them all. \n",
+ "The coursework is structured so that the code must be developed in a series of steps, each building on or incorporating its predecessor. There are six functional requirements (FRs) as follows:\n",
"\n",
+ "| FR | Description |\n",
+ "|-----|-----------------------|\n",
+ "| FR1 | Arithmetic mean | \n",
+ "| FR2 | Read column from file | \n",
+ "| FR3 | Read file | \n",
+ "| FR4 | PCC for two lists | \n",
+ "| FR5 | PCC for file | \n",
+ "| FR6 | Print table | \n",
"\n",
- "* one problem I had with FR3 was discovered when using the function in various of FR4. The original dictionary produced string key-value pairs - which when printing looked fine to me. It took me some time to realise that the values were string - that they were enclosed in single quotation marks. This was certainly a source of frustration - when something simply was not behaving the way I was anticipating.\n",
- " * However, once I noticed, I tried to amend the list comprehension by convernting to integers with int() - which caused an error - \"invalid literal for int() with base 10: '60.5'\"\n",
- " * Therefore I converted the data to floats - which may not be quite right for ages, but will not impact the calculations.\n",
- "* \n",
"\n",
- "FR4: Pearson Correlation Coefficient\n",
+ "Pearson's correlation coefficient measures linear association between two variables, ranging from -1 to 1, where -1 is perfect negative linear correlation and 1 is perfect positive linear correlation. The formula for calculating the PCC is as follows:\n",
"\n",
- "* I spent a lot of time on this function and tried to make it work using several methodologies. \n",
- "* Pseudo code was particularly helpful in this instance, but I have a tendency of starting with pseudo code and then deviating into a confused space of trial and error rather than updating the pseudo code. Definitely something I can work on and take away into my current day job. \n",
- "* Firstly I needed to remind myself how to calculate Pearson's r having only have had it calculated for me in SPSS or R. There a couple of approaches - but the strategy is to break down the formula into standalone calculations and build the function to produce these before putting it all together to return R. \n",
- "* I spent a lot of time trying different approaches. Again, like last time I started with for loops and enumerate() but found that I couldn't always achieve my intentions - that I would get slightly lost in the hierarchy and multilined approach, so I used list comprehensions instead and in the end, I am pleased with the result.\n",
+ "Given paired data, consisting of $n$ pairs: \n",
+ "$$\n",
"\n",
- "* As previously, I tried thinking about generalising and future-proofing the function for future use and that includes considering what tests / checks / validation I could put in place. The most obvious source of error will be the csv file so I added some assertions to ensure that the csv file has the correct data before proceeding to calculate Pearson's r. \n",
- "* I tested my version of Pearsons with numpy corrcoef() and it produces the same results for a small data set, which gives you that excitement of having achieved your intentions.\n",
+ "r =\n",
+ " \\frac{ \\sum_{i=1}^{n}(x_i-\\bar{x})(y_i-\\bar{y}) }{%\n",
+ " \\sqrt{\\sum_{i=1}^{n}(x_i-\\bar{x})^2}\\sqrt{\\sum_{i=1}^{n}(y_i-\\bar{y})^2}}\n",
+ "$$\n",
"\n",
"\n",
- "FR5: Pearson Correlation Coefficients for a file\n",
+ "source: [Wikipedia](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient#Definition)\n",
"\n",
- "* This function builds on FR4 by calculating Pearson's r for each pair of columns in a csv file.\n",
- "* I enjoyed using FR3 and FR4 (which in turn use FR1) within this function. That is pretty cool.\n",
- "* It does not have many lines of code but it took me a while to get it working. I had to think about how to iterate through the columns of the csv in terms of what has been returned by FR3 and FR4 and then getting the column names into a list of tuples.\n",
- "* It was not plain sailing and there was a fair bit of frustration - another good example of where I knew exactly what I wanted to do but it was hard to get there. I had a few versions on the go - I think I could have benefited by sticking with one and updating the pseudo code. \n",
- "* As previously, I moved from for loops to list comprehensions as I had better success getting the results I wanted, especially returning lists of tuples. Initially, I was returning individual lists instead of lists of tuples. \n",
- "* The joy of success after some struggle can be very satisfying. \n",
"\n",
- "I would probably look to round some of the output in future iterations.\n",
- "Actually - I just added the rounding now!\n",
+ " \n",
+ "### <b> Reflection on Development Process</b>\n",
"\n",
- "And finally\n",
+ "My development process made use of the inherent stratified nature of the task, allowing me to plan, develop and test each FR independently in separate code cells, before combining where appropriate. I found ths particularly useful for the more complex FRs - 4, 5, and 6 which required significant iteration, investigation and testing before achieving the desired results. This also worked very well in terms of source and version control, complementing the use of Git.\n",
"\n",
- "FR6: printing the results\n",
+ "My strategy was to use a localised crisp-dm approach which included initial articulation of requirements followed by iterative pseudocode, Python code and testing cycles until the desired results were achieved. I find this approach very effective and it is one I use in my current employment. However, having said that, I find that I can occasionally spiral out of control in the testing phase, which can be time-consuming, frustrating and ultimately less productive. I will continue to try to be mindful of this. \n",
"\n",
- "* I found this FR very challenging and a bit frustrating.\n",
- "* I could have benefitted significantly by sticking with my pseudo code and keeping it up to date - but once I got stuck into a cycle of trial and error, minor success and subsequent set-back, I did not rever to pseudocode and focused on cracking the issue which was hampering me. \n",
- "* This function was created through trial and error - entirely\n",
- "* I broke it down into separate sections and tried to think how I would iterate through the list of tuples to get a decent end result.\n",
- "* Some sections worked, others didn't, changes then impacted the rest of the function and I was back to square one - hardforking!\n",
- "* Towards the end, I had functioning portions - the top table header, the column headers, the rows headers, the r-values. This was a mess of for loops, list comprehensions, and string formatting which I needed to work through step by step.\n",
- "* I found cells in Jupyter notebooks to be invaluable - the ability to run portions of code and easily move it around is very effective and powerul. I also found the ability to comment out sections of code very useful - I could easily comment out sections of code and then uncomment them to see how they impacted the rest of the function.\n",
- "* Finally, the pieces (rows of code) were lining up - at this point, I resorted to tweaking and assessing the impact of each change. I discovered that I needed data specific to the input table - rather than fixed widths - so created a separate function to calculate max column widths and then played with that to get the table to look passable. \n",
"\n",
- "Whew - got there in the end! \n",
+ "[create and insert a crisp-dm diagram here]\n",
"\n",
- "Although this was more challenging to get motivated over as I am sure I would use some library to print tables without the effort and challenge - i.e. pandas, prettytable, etc. However, in hindsight I am glad I persevered as I have learned a lot about string formatting and how to iterate through lists of tuples. It allowed me to trouble-shoot, iterate through a requirement, reassess and rework through many cycles. I am sure I will use this knowledge in the future.\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "add markdown text here\n",
+ "I tried to be disciplined in my approach and for the most part feel like I was successful. I made deliberate attempts to make use of newly acquired tools and techniques like Git, VS Code, Jupyter notebooks, Markdown - ensuring that I commented on my code, made use of Markdown cells during the development phase, and committing changes to Git in appropriate intervals.\n",
"\n",
- "coding - use of x, y, common practices, etc.\n",
- "naming conventions (camel, underscore, etc.)\n",
- "level/amount of commenting\n",
- "efficiency of code\n",
"\n",
+ "### <b> Code Evaluation </b>\n",
"\n",
- "Thought I was attentive to detail\n",
- "Eg print statements \n",
+ "When it comes to code evaluation, I think it is important to acknowledge that the Task_1 code has been developed to meet the coursework specification. As such, it has been produced with certain limitations and constraints and does not necessarily reflect my approach in a different context, such as my employment or personal projects. However, it is important to reflect on what has been produced. \n",
"\n",
- "Challenging to get the iteration done\n",
- "Know what I want to do but achieving it programmatically is hard and slow\n",
+ "Overall, I am pleased with the code that I have produced. It achieves the requirements (as I have interpreted them) and it <i>feels</i> fairly efficient and robust. \n",
"\n",
- "Balancing the need to know vs how to find out what I need to know\n",
- "For example researching code, assessing, evaluating, understanding, not assuming. \n",
+ "I tried to be mindful of certain principles when undertaking this work:\n",
"\n",
- "Knowing fundamentals but not everything. Realistically would use libraries where they do what you need them to. Maybe not most efficient, but most practical or ease of use or pretty. Maybe not quickest run time but quicker to get up and running. There's a balance to be struck - knowing what your requirements are and adapting to them...metaphors I use - chefs and recipes and ingredients and cake shops. Sometimes you need to buy the cake pre-made and decorated, sometimes you make it from scratch, sometimes you buy it and decorate it, sometimes you farm the eggs. \n",
+ "* Future-proofing the functions, i.e. making them flexible and adaptable to different data types and formats, perhaps accepting different parameters, etc. \n",
+ "* Thinking about likely pitfalls from the users perspective and pre-empting likely errors with assertions and exception handling\n",
+ "* Unambiguous, self-explanatory naming of functions and variables, where possible\n",
+ "* Balanced comments and docstrings - considering approaches like DRY (Dont Repeat Yourself), WET (Write Everything Twice), KISS (Keep it Simple, Stupid) bearing in mind possible readers of the code\n",
"\n",
- "Or building a shed: Building shed analogy \n",
+ "I think I have been reasonably successful in achieving these goals. I have tried to make the code as flexible as possible, accepting different parameters and data types, and I have tried to make the code as robust as possible, using assertions and exception handling to pre-empt likely errors. I have saved alternate versions of the functions for future use and reference, which take additional parameters or employ a different approach. \n",
"\n",
- "Don't need to know how to build the pieces individually\n",
- "Carpentry power tools skills\n",
- "May get a better product\n",
- "May get exactly what I want\n",
- "But requires a sig investment and aptitude whereas can by prefab, deconstructed where hard work done. You can focus on the outcome and the nuance and styling\n",
+ "In terms of naming convention, documentation and comments, I think I struck the right balance. If anything, I have probably over-commented slightly but this is because I am erring on the side of caution. I have yet to settle on a naming convention and standard and find that I flipflop between different methods like camelCase, snake_case, PascalCase, etc. I am also aware that there seem to be conventions for variables and functions, especially iterables, which I am not fully using. \n",
"\n",
- "Or decorating a room - v being interior designer\n",
+ "As for the code itself, I think it is fairly efficient and robust. Of course, there is always room for improvement and I would very much like to know of any suggestions or improvements, or better approaches.\n",
"\n",
- "Coding and documenting. Importance of. Dry wet. Balance. "
+ "\n",
+ "#### <b> Summary </b>\n",
+ "\n",
+ "Whilst I have a background in elements of data and systems, I am only getting started in the programming part of this journey. I am super keen to learn for my personal and professional development and want to pick up best practices, adopt standard approaches and avoid pitfalls and repeating mistakes. When it comes to Python, I am amazed at how many different ways there are to solve a particular scenario. This can make it impossible to identify a 'single best approach' - something I will need to get used to and embrace. \n",
+ "\n",
+ "\n"
]
},
{