Thread by @dcuthbert, This is a thread for @Matt_Gerlach on how one could better work [...]

This is a thread for @Matt_Gerlach on how one could better work with data collected from pihole. However, it could also be useful for anyone else who wants to better understand how pervasive the global tracking world is and to do something about it. #privacy #surveillance.

First up, adblockers do not work anymore. The industry has moved on a lot (they use the same ones you do, don't kid yourself that this industry isn't blackhat af and do dodgy thing)

It's better to cut the snake's head off rather than make it dance to your beat.

Tools you need:

1: https://pi-hole.net/
2: raspberry pi (any will do, just needs some network interface
3: blocklists, you can use mine to start https://github.com/danielcuthbert/trackers

danielcuthbert/trackers

A collection of online trackers, advertising networks, telemetry collectors and general 'we like to watch' lists for you to use to get better control on what they know and can see. ...

https://pi-hole.net/

Now the best thing is to create a baseline of what you browse over a period of time. I usually say 30 days as this gives you a nice spread.

Once you've done that, your DB will look pretty cool. You've got two key things here: a web interface

Or the source of that, which is an SQLite3 database, found at /etc/pihole/pihole-FTL.db

So first you want to back that up and get it onto a box where you can manipulate it

sqlite3 /etc/pihole/pihole-FTL.db ".backup /home/pi/pihole-FTL.db.backup"

Once you've got that, I'd recommend using DB Browser for SQLite. It should look like so

The key tables you want are:

- network_addresses (where all the queries come from if you have a wide network with numerous sources)
- queries (the most important table)

Now the latter will be large and this could prove problematic, so you could prune this a little.

Now you can do all of this via the web UI and it's very good but you could also use the command line. Say I wanted to see all requests from one of my APs

SELECT DISTINCT domain FROM queries WHERE client IS "192.168.0.28"

For more info on the table structure, see

https://docs.pi-hole.net/database/ftl/

If you just wanted to show all blocked queries (status 1) then

SELECT DISTINCT domain FROM queries WHERE status is 1

then export this as CSV and use something like

https://app.rawgraphs.io/

Now this is where you'd work out what you want from visualisation. Say you want to show all blocked domains?

Or if you want to get fancy, take blocked domains and then show frequency for each domain. It all depends on what exactly you want to do.

For most, the pihole UI is very good so i'd recommend using that at first and fine-turning your block lists based off your own baseline.

Now creating blocklists. There's many a way you can do this, you can use ones already made by people or you can explore it yourself. The latter is one approach I prefer.

Let's take a traditional web app, say a dirty ugly news website

Developer tools are your friend here. What you want to do is inspect every single element, such as what is being loaded (JavaScript), endpoints being called etc.

Turn on developer tools (found in the menu of the browser)

Here's one for adverts called by mol-adverts.js via XHR. So we don't like this, so adding it to our own block list.

Basically, rinse and repeat is the name of the game here.

But what happens if you have a mobile app? How do you do the same?

With iOS you can attach a remote developer tool session via safari but it's a hassle, what I prefer doing is pulling down the mobile app and then running it through https://github.com/MobSF/Mobile-Security-Framework-MobSF

Basically, MobSF is an all-in-one automated mobile application pen-testing framework. Yes, mostly for those looking for vulnerabilities or bugs but also something you can use to find out what that app is doing.

You need a few things here:

1: docker (think of this as a platform for creating and running containerised applications) https://www.docker.com/why-docker

Why Docker? | Docker

Learn why Docker is the leading container platform — Freedom of app choice, agile operations and integrated container security for legacy and cloud-native applications.

https://www.docker.com/why-docker

Once you have it installed, go to your terminal and issue this

docker pull opensecurity/mobile-security-framework-mobsf

this will pull down the latest version of MobSF.

The next thing you'll need is the mobile app file in question. For Google it will be an APK and for ios an IPA.

You can use https://ipa.rocks/ and https://apkpure.com/app

Just add the name and pull it down

Download APP APK Android App Online - Free Pure APK Downloader

https://ipa.rocks/

Word of caution: this stage is a little bit advanced and can be very daunting to those who aren't familiar with it. But, ask questions and experiment, you can't cut yourself or blow stuff up.

The command you want to run is this

docker run -it -p 8000:8000 opensecurity/mobile-security-framework-mobsf

basically tells it to run it, listen on port 8000 and use the latest image you pulled above.

If all went well, you should see this in the terminal and when you visit http://0.0.0.0:8000, you should see

Ok so what exactly are we looking for here? am I hacking? do I get to wear a hoody and have chains on my keyboard?

Not exactly, you are peeking behind the curtains to see what this application is doing. The bits we are interested in are endpoints (servers that the app chats to)

So we drag our apk or ipa onto the file and let MobSF do its magic.

Remember we are looking for trackers here. Trackers are basically services that monitor what you do.

Now this will take around 10 minutes to work. I do this for every single app my kids use. I don't want to stop them from using the apps but I do want to understand how that app was developed, who it speaks to, what it collects and so on.

Once it is done, you will be presented with a screen like so. This is a lot of information to handle, so ignore most of it and got to the left and look for reconnaissance and click trackers

Here we have three that are of some concern:

braze
conviva
google firebase.

the top two are well-used and track a lot. For example

Now the cool thing is that you can see the endpoints in use here:

http://appboy.com
http://cws.conviva.com http://firebase.com

Firebase

http://appboy.com

Once you have these, head on over to your pi

group management --> domains

Add each of these into the field and make sure you tick "Add domain as wildcard"

Congrats, you've just stopped the app from being able to call home.

Now, this might seem rather longwinded and indeed complex but sadly stopping tracking is a cat and mouse game.

Hope this helps someone. As always, ask questions!!

@threadreaderapp unroll

Twitter isn't the best place for such information, so I've turned this into a markdown page and made it accessible over at https://github.com/danielcuthbert/trackers/blob/master/HOWTO.md

TODO: Play with https://datasette.io/ (thanks @snkhan for that) and look at how pihole might be extended to better extract info

danielcuthbert/trackers

A collection of online trackers, advertising networks, telemetry collectors and general 'we like to watch' lists for you to use to get better control on what they know and can see. ...

https://github.com/danielcuthbert/trackers/blob/master/HOWTO.md

You can follow @dcuthbert.

Tip: mention @twtextapp on a Twitter thread with the keyword “unroll” to get a link to it.

Latest Threads Unrolled: