Thanks for sharing the design, I however have a question. How do you horizontally scale after sharding? You will have a new shardId, but
1. Do you readjust the ids of all the existing adjust so that the data is equally distributed again. But that means processing all data and if the ids are interacted outside world (urls as in your case?) its impossible.
2. You don’t re-adjust, in that case latest shard will be relatively empty. Do you change the probability on the basis of disk space when giving shardId to a new object?