Just because you can do something doesn't always mean you should. Embedding all of your application logic directly in the database can make tracking migrations and releases difficult. At the same time, a complex pipeline that takes a nightly extract, loads something into Spark, generates results, that you then feed back into the database isn't exactly lightweight. In the case of plpython3u and pandas, scheduling something like the above to run daily with pg_cron could be a much simpler solution. With a mix of SciPy, NumPy and Pandas there is a lot of interesting potential here and I'd love to hear what practical uses others come up with @crunchydata, or give it yourself a try-our database-as-a-service Crunchy Bridge comes already preconfigured with plpython3u and SciPy, NumPy, and Pandas.

PYTHON data analysis using the Pandas library to manipulate datasets and automate tasks from Excel practical application. In this course, I will help you to simplify and automate your data analysis and data science tasks using the Python and the Pandas library. These lectures are the result of my personal crash course in Python programming learning experience. I have recently changed jobs and have had the opportunity to learn Python programming to analyse and manipulate data. I have compiled some essential techniques as well as tips to make sure you understand how Python object-oriented programming works.

Sometimes the sample data that data scientists are given does not fit what we know about the wider population data. For example, lets assume that the data science team were given survey data and we noticed that the survey respondents were 60% male and 40% female. In the real world the UK general population is closer to 49.4% male and 50.6% female (source: https://tinyurl.com/43hpe5e4) There could be many explanations for our 60% male sample data. One possibility is that the data collection method might have been flawed.