//Thread on how twitter timeline architecture works technically//

Twitter timeline: Only show those tweets in which the user is highly interested in.

Twitter doesn't show all of the tweets tweeted by users whom you're following.

#100DaysOfCode #MachineLearning

2.

How twitter filters out only engaging tweets out of all tweets? How twitter gets to know this. Machine learning is the answer.

How does the twitter timeline architecture look like?

Is it micro-service or monolith architecture?

Read

Twitter uses micro-service architecture to solve this problem.

There are various sub-systems involved in this that interact with each other.

Full pipeline:

1. Requesting tweets: First, the client requests for tweets. This happens whenever a user logs into his Twitter account.

2. The request is sent to the tweet service.

3. Tweet service forwards the request to the candidate service. The role of candidate service is to fetch all of the possible tweets at any particular time for that user.

4. Then, all of the candidate tweets are returned to tweet service as the response.

5. All of those candidate tweets are then send to prediction service by tweet service.The role of the prediction service is to score the tweets based on engagement using machine learning.

6. Candidate tweets with scores are then sent to tweet service.

7. Tweet service then sorts those tweets based on the score in decreasing order and then return those tweets to client/user.

8. The tweets which are finally shown to user are stored as the training data for the re-training of the machine learning model in prediction service.
And the real-time engagement that the user had with those tweets are stored as true_labels for the data.

Many questions need to be still explored though:
- How candidate tweets exactly works? How it fetches all of the tweets, especially the tweets that are liked by those users whom the client is following?

- How does a machine learning model/architecture of prediction service look like?

- What database does twitter use for collecting future training data?

- How engagement is collected and then send over to tweet service in real-time?

- How these services communicate with each other? Through APIs or something else?

- What this service so fast to run in production enviroment?

I'm on the journey to explore and learn about how twitter works technically. Will be posting all of my learning notes here publicly on twitter and as well on my blog.

Read full blog post: https://rishabhgarg.dev/how-twitter-timeline-architecture-works

Source: https://conferences.oreilly.com/artificial-intelligence/ai-ny-2019/public/schedule/detail/73559

Slides by @satanjeev