A few years ago Pinterest released their own Search Framework (Manas) in order to support their growth.
As they use Manas for the majority of their products, they made it a high priority matter to improve its indexing latency.
Manas already supports incremental indexing, which is able to provide indexing latency within the order of tens of minutes.
Michael Mi wrote a super insightful story about the approach they took to solve the problem. It contains tons of details on the architecture, failover and auto-scaling, error recovery, etc.Read more...