Designing a News Feed Service using Document DB as a storage
- Chayanika Misra
- Aug 3, 2024
- 3 min read
Design a News Feed service to quickly generate and fetch the news feed and profile feed for each user present in the social network.
Model design
User table (to store user info)
Followers table (to store following id(foreign key to user id) and follower id(foreign key to user id))
Post table (to store posts and user_id(foreign key to user id))
Fetching the news feed
To generate the news feed for user user1 when the user logs in to view his/her news feed
Fetch all users user1 follows and then fetch their posts and present it as a news feed.
To do this, we join followers table with the user table on user_id to fetch the user user1's followings users and then join on posts table for all followings users of user1 to get their posts.
Problems with the initial design
Fetching all posts when a user logs in very time consuming, specially if the user follows plenty of other usersGenerating the news feed for a user this way also involves multiple queries (3 queries in our case)
Solution
How companies like Twitter solve these two problems is by pre-computing the news feed for each user. The pre-computing is done using a write fan out model (So when a user posts a post, the post gets appended to the feeds of the user’s followers)
Then the pre-computed news feed can be stored in a cache for the user. This ensures whenever the user logs in the news feed is already generated and read time access is much faster.
Problems with this design
Lets first discuss the problems of using a cache to store the news feed for users.Even though this solution scales, problems of using a cache is that of limited memory, because cache memory is much more expensive compared to disk.And to solve for single point of failure, you would need multiple replicas of the cache, that would add to the cost.
How Document DB fits in this design
We can overcome the above problems by using a NoSQL DB to store the pre-computed news feed.
Why NoSQL over SQL?
Because, we want a flexible schema and we care only about the scalability here, we can compromise on the consistency and have a eventual consistent design.
Why Document DB over other NoSQL DBs?
Because document databases have a flexible schema, they can store documents that have different attributes and data values.
Using a document database, you can store each user’s news feed and profile feed efficiently. Suppose that a user elects to add or remove a new post from their profile or news feed. In this case, their document could be easily replaced with an updated version that contains recently added posts to the news feed. Document databases easily manage this level of individuality and fluidity.
Mongo DB and CouchDB are both good choices of document NoSQL DBs, major difference being MongoDB stores data in BSON format (a variant of JSON that stores data in binary format) while CouchDB stores data in JSON format.
For your reference:
Blog on getting started with Couch DB in your spring boot project
here’s an implementation of a News feed generator project using above write fan out logic with Couch DB and Spring Boot
Just exploring the high level design space and implementing services my way. Do let me know in the comments if you feel there can be a better design for a news feed service.For any queries, reach me at chayanikamisra1997@gmail.comThank you!!
Comments