A Beginner’s Guide to learn web scraping with python!
Posted by Armando Brito Mendes | Filed under lições, materiais ensino, materiais para profissionais, software
Boa descrição de web scraping com Python
Web Scraping with Python
Imagine you have to pull a large amount of data from websites and you want to do it as quickly as possible. How would you do it without manually going to each website and getting the data? Well, “Web Scraping” is the answer. Web Scraping just makes this job easier and faster.
In this article on Web Scraping with Python, you will learn about web scraping in brief and see how to extract data from a website with a demonstration. I will be covering the following topics:
Tags: Python
Build Pipelines with Pandas Using pdpipe
Posted by Armando Brito Mendes | Filed under linguagens de programação, software
Boa descrição de pipelines com os data.frame do Pandas.
Pandas is an amazing library in the Python ecosystem for data analytics and machine learning. They form the perfect bridge between the data world, where Excel/CSV files and SQL tables live, and the modeling world where Scikit-learn or TensorFlow perform their magic.
A data science flow is most often a sequence of steps — datasets must be cleaned, scaled, and validated before they can be ready to be used by that powerful machine learning algorithm.
These tasks can, of course, be done with many single-step functions/methods that are offered by packages like Pandas but a more elegant way is to use a pipeline. In almost all cases, a pipeline reduces the chance of error and saves time by automating repetitive tasks.
In the data science world, great examples of packages with pipeline features are — dplyr in R language, and Scikit-learn in the Python ecosystem.
A data science flow is most often a sequence of steps — datasets must be cleaned, scaled, and validated before they can be ready to be used
Following is a great article about their use in a machine-learning workflow.
Managing Machine Learning Workflows with Scikit-learn Pipelines Part 1: A Gentle Introduction
Are you familiar with Scikit-learn Pipelines? They are an extremely simple yet very useful tool for managing machine…
Pandas also offer a .pipe method which can be used for similar purposes with user-defined functions. However, in this article, we are going to discuss a wonderful little library called pdpipe, which specifically addresses this pipelining issue with Pandas DataFrame.
In almost all cases, a pipeline reduces the chance of error and saves time by automating repetitive tasks
Tags: data mining, pandas, pipelines, Python
The Beautiful Hidden Logic of Cities
Posted by Armando Brito Mendes | Filed under mapas SIG's, materiais ensino, materiais para profissionais, visualização
clicar na imagem para seguir o link
Padrões identificados em mapas de cidades.
After finishing my map of the most common road suffixes by length, I realized I could also map each individual road, colored by its suffix. This has led to the loveliest maps I’ve made.
Driving around your city, you’re probably somewhat aware of Avenues and Boulevards and Streets and Roads and so on. Here in Portland, at least, I know that Avenues run north-south and Streets run east-west. However, it’s hard to get an overall view of how all these road designations knit together. By coloring them, we can suddenly see a new, stunning view of what we normally take for granted.
Tags: captura de conhecimento, image mining, mapas
Machine Learning and Data Science Cheat Sheet
Posted by Armando Brito Mendes | Filed under Sem categoria
clique na imagem para seguir o link
Pequeno tutorial (com muitos links) sobre linux e machine learning
You can download the new machine learning cheat sheet here (PDF format, 14 pages.)
Originally published in 2014 and viewed more than 200,000 times, this is the oldest data science cheat sheet – the mother of all the numerous cheat sheets that are so popular nowadays. I decided to update it in June 2019. While the first half, dealing with installing components on your laptop and learning UNIX, regular expressions, and file management hasn’t changed much, the second half, dealing with machine learning, was rewritten entirely from scratch. It is amazing how things have changed in just five years!
Tags: data mining, Linux, machine learning
Usage of Asterisks in Python
Posted by Armando Brito Mendes | Filed under linguagens de programação
clique na imagem para seguir o link
Um tutorial sobre os vários usos do * no Python
Most of us use asterisks as multiplication and power operators, but they also have a wide range of operations being used as a prefix operator in Python. After reading this article, you will get to know the full usage of asterisks.
Asterisks have many particular use cases in Python. In general, we are familiar with the multiplication and power operators. It can perform some other operations like unpacking, arguments passing, etc.., in different situations. First, let’s see the general usage of asterisks.
Tags: Python
The Sleep Blanket
Posted by Armando Brito Mendes | Filed under visualização
clicar na imagem para seguir o link
A visualization of my son’s sleep pattern from birth to his first birthday. Crochet border surrounding a double knit body. Each row represents a single day. Each stitch represents 6 minutes of time spent awake or asleep #knitting #crochet #datavisualization
Tags: belo
Making of the Illustrations of the Natural Orders of Plants
Posted by Armando Brito Mendes | Filed under materiais para profissionais
clique na imagem para seguir o link
If someone told me when I was young that I would spend three months of my time tracing nineteenth century botanical illustrations and enjoy it, I would have scoffed, but that’s what I did to reproduce Elizabeth Twining’s Illustrations of the Natural Orders of Plants and I loved every minute.
After the unexpected successes of my Byrne’s Euclid and Werner’s Nomenclature of Colours projects (for which I’m very grateful) I got the itch to follow them up with another reproduction of an obscure catalog from the 1800s. However, finding interesting obscure catalogs want an easy task when I didn’t know what would pique my interest. Anything was fair game but I had an inkling that something based on the sciences would be most interesting. Scientific catalogs are organized, structured, and data can be extracted from them with some elbow grease.
Tags: belo, captura de conhecimento, data mining, machine learning
Posted by Armando Brito Mendes | Filed under estatística
clique na imagem para seguir o link
Bom texto com conclusões exclusivas
1. Re-sampling and Statistical Inference
- Main Result
- Sampling with or without Replacement
- Illustration
- Optimum Sample Size
- Optimum K in K-fold Cross-Validation
- Confidence Intervals, Tests of Hypotheses
2. Generic, All-purposes Algorithm
- Re-sampling Algorithm with Source Code
- Alternative Algorithm
- Using a Good Random Number Generator
3. Applications
- A Challenging Data Set
- Results and Excel Spreadsheet
- A New Fundamental Statistics Theorem
- Some Statistical Magic
- How does this work?
- Does this contradict entropy principles?
4. Conclusions
Tags: data mining, inferência, machine learning
Map of Best Breweries in America
Posted by Armando Brito Mendes | Filed under Investigação Operacional, mapas SIG's, visualização
clicar na imagem para seguir o link
Um mapa com as melhores produtoras de cerveja artesanal nos EUA e uma rota otimizada com algoritmos genéticos
RateBeer puts out a list every year for top 100 breweries in the world. The rankings are based on reviews, range across styles, and historical performance (and maybe a bit of subjectivity). RateBeer just published the list for 2018. Here’s a map of the 73 U.S.-based breweries.
Brewery Road Trip, Optimized With Genetic Algorithm
Now that we know where they are, let’s find out how to visit all of them in one go.
Tags: belo, grafos, mapas, otimização, R-software, SIG
Eurostat – Eurpean Statistics
Posted by Armando Brito Mendes | Filed under data sets, estatística
clique na imagem para seguir o link
