I'm currently scoping what it would take to improve the latency in the replication feature of Accumulo. I'm interested in knowing what work, if any, is being done to improve replication latency? If work is being done, would there be some interest in collaborating on that effort?
If nothing is currently being planned, I'd be interested in design ideas and pointers from the community for improvements to the existing implementation. We're looking to get replication down to less than five minutes and are willing to put in the effort to implement the improvements.
Thanks, Josh. I think the main pain-point is that replication doesn't occur until the WAL is closed. We've made some aggressive configuration changes to Accumulo to reduce the WAL time rollover and minor compaction frequency to force replication to go faster. It is down to around 20 minutes or so on our production clusters, but we are kind of at our limit starting to affect our query performance.
My initial thoughts are to increase the replication parallelism and start replicating the WAL before it is closed (I see a few JIRAs open already that mention these things), but I haven't done enough digging in the code base to see what is really available.
Are you free for a bit in the near future to meet up for a bit and talk replication? I'll buy lunch!
On Wed, Feb 15, 2017 at 2:52 PM, Josh Elser <[EMAIL PROTECTED]> wrote:
Gotcha. That's definitely the biggest factor that I was aware of. I wasn't sure if you knew more than I did by now ;). I can respect the implications of too much bookkeeping going on. That might really start pounding the metadata and replication tables.
Happy to do lunch, also happy to just have a video call too if that's more convenient.