Countif and Sort, 1 million times faster

From time to time, I got to deal with CSVs that are millions lines each, whether the data is from large datasets or from web scraping.

One time, I extracted the followings of the followers of an account, having in mind to know with which accounts they were competing on their audience.

That was an 10k followers account - and, give or take, each follower was in average following 800 accounts as well: that's 8,000,000 lines in total. And I had this data in a CSV, not in a database (oup's).

Question is: what are the accounts the most followed by the follower of an account?

That's a basic COUNTIF question.

Excel only takes into account a little more than 1,000,000 rows. Even more - try to perform a =COUNTIF() on 1 million rows. You'll get the result two days after.

So here I'm just sharing a little Python script, that actually does the work in less than 2 minutes. I'm leaving comments to explain the script below.

Hope it'll help someone!

import csv
import collections
id_followed = collections.Counter() ## we initialize the COUNTIF column as a Counter
outFile = open('occurences.csv','w') ## first output file
outFileSorted = open('sorted.csv','w') ## second output file, sorted
listLines = []
## the output-followings.csv file structure was:
## id_follower, id_following, username_following
## 123456, 7890987, following1
## 123456, 0987890, following2
with open('output-followings.csv','r') as input_file:
   for row in csv.reader(input_file, delimiter=','):
       print(row[2]) ## prints in console the username
       id_followed[row[2]] += 1 ## increment in the Counter corresponding to the username
with open('sorted.csv','w') as sortedfile:
    fieldnames=['followed','occurences']
    writer = csv.writer(sortedfile)
    writer.writerow(fieldnames)
    for key, value in id_followed.most_common(1000): # most_common() gives us the possibility to directly sort the Counter
        line = str(key) + ',' + str(value)
        writer.writerow(line.split()) # we write in the sorted.csv file
with open('occurences.csv','w') as csvfile:
   fieldnames=['followed','occurences']
   writer = csv.writer(csvfile)
   writer.writerow(fieldnames)
   for key, value in id_followed.items():
       line = str(key) + ',' + str(value)
       writer.writerow(line.split()) # we write all the rows in the outFile
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

>

Get the right insights for your brand from your data & analytics

Measure what matters - the rest doesn’t count.

Measure what matters.

Growth Insights • More articles

Your CRM action plan to take your ecommerce store to the next level [checklist included]

Launching an ecommerce store? You have everything ready, your products, your website, your design - even your analytics? Yupee !But how about your CRM ? In this article, we'll review a simple go-to plan you can start with today to extend your CRM capabilities. Let's dive in!

Read →

The Perfect Email Journey to get back +25% of your lost checkouts

I'm sharing here a proven tactic that drive results to get back at least 25% of your lost checkouts, while the e-commerce average is around 8%.

Read →

Facebook Ads Budget & Profitability Calculator

This calculator will let you know which budget you'll need on Facebook - and if your ads are profitable.‍

Read →

Emails Metrics A/B Test Significance Test

This email A/B test significance test will let you know which version of your email A/B test is the most performant - and on which metrics. Plus, we'll display which recommandations you can take.

Read →

Funnel Review / Colourpop

Colourpop is a great makeup and skincare brand. It is well-known for its cosmetics and well-appreciated by beauty addicts around the world. We were curious how they gained even more popularity, so we did what we love to do - review their funnel. Let’s see all their good practices and what they could improve on their website!

Read →

5 eCommerce Marketing Strategies You Could Implement (Right Now)

The global eCommerce industry is pegged at a whopping $4.89 trillion in 2021, according to Shopify. If you get it right, it’s a profitable and borderless industry that you could take advantage of. 

Read →

Funnel Review / Peloton

Peloton is a brand well-known for its exercise bikes, treadmills and fitness classes. Take a ride with us as we get a closer look at their funnel. Highlight vendor collections - The home page is the most important part of your site. It helps...

Read →

How to Create SEO-Optimized FAQs To Get More Traffic

SEO-optimized FAQ for your eCommerce store gives people the information they're looking for without them having to leave your website or call customer service.

Read →

What is a Good Conversion Rate On Shopify? [& Tips On How to Improve Conversions]

To help boost your Shopify store conversion rates, you’ll need a holistic approach.

Read →

The Perfect Product Page Structure for Your Ecommerce Store

The Product Page is the most crucial page of your eCommerce store where conversion takes place.

Read →

How to Use Pinterest To Promote Your eCommerce Store

Pinterest is a great platform to promote your eCommerce business.

Read →

eCommerce Personalization Examples & Tactics For Shopify

Personalizing your eCommerce site is a great way to improve conversion rates.

Read →

Growth Blog • D'autres articles

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
▲ hutte • que 2021 soit spécial • réalisé avec attention