Bringing together the AI community in Singapore – companies, startups, researchers, students, professionals – to collaborate, find research and business opportunities and talent.
- Have an interesting story to share?
- Seeking for AI talent for your organization?
- Seeking research interns for your labs?
- Seeking an industry partner for your AI projects?
- Are you a researcher and seeking an industry partner to do a POC or deployment of your IP/research outcomes?
Intermediate Python for Data Science
Do you have any questions relating to Intermediate Python for Data Science? Leave them here!
In part 1 Matplotlib - Scatter Plot (1)
I see plt.xscale('log') being introduced to compress gdp_cap on x-axis when scatter plotting life_exp on y-axis.
This got me questioning .
When plotting a y=log(x) graph (or a scatterplot close to that) , why do people look for straight lines (to argue there is good correlation) after doing things like transforming x axis to log(x) and relabeling/compressing the xtick positions of x values of 1,2,3,4 ..... so on. Isn't that lying to yourself?
By the same reasoning, we can transform any non linear function of y=f(x) into a straight line visually by manipulating the x tick positions right? Just to say "we have a straight line"
Going from visual tick manipulation to math, does Universal approximation theorem apply to 1 input - x and 1 output - y scenarios? If yes, can we then always say y is a linear function of combination_of_transformations (x)?
What are the special advantages of being able to visually see/mathematically transform to straight lines that gives linearity such high prominence in math/statistics? Why not leave things in their non-linear form?
Is it a limitation of people's minds that we cannot make confident decisions until we visually see linearity? (just like some people can draw perfect circles, i believe there exists people who can write the equation of a log graph with the correct base and scaling just by looking at the graph, and they would be perfectly comfortable with not looking at straight lines)
On transforms, I can see sin, cos, tan came naturally from circles and triangles, and calculus came naturally from trying to make sense of rates of change, did transforms like log (don't mean base e), exponential, arise in nature? Are there examples of other transforms yet to be invented or we have discovered all possible? (meaning all mathematical modeling is already constrained/settled to an established vocabulary of basic transforms)
One answer is because by establishing linear relationship, it opens up the possibility of applying certain families of algorithms for analysis without breaking any assumption, e.g. Generalised Linear Model, and to a certain extent, SVM and PCA. These algorithms are usually, in some aspects, simpler for modelling (e.g. as opposed to do non-linear modelling), and hence preferred. We can use the transformed variable in place of the raw variable to maintain linearity.
On the second question, I doubt we have discovered all possible "natural" transforms.
Hope this helps.