HBASE-29796 Allow sleepForRetry replication config to be overridden by replication peers

[sidkhillon]

Currently, sleepForRetries (the sleep time between retry attempts during replication) is only configurable globally via the replication.source.sleepforretries configuration property. This makes it impossible to tune behavior for individual replication peers that may have different requirements.

This change would add support for configuring sleepForRetries on a per-peer basis, with fallback to the global configuration when not set. It also adds UI support for displaying the field and shell support for editing the value.

This is related to #7578 because this will not cleanly merge into branch-2

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 27s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	buf	0m 0s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+0 🆗	mvndep	0m 20s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 3s		master passed
+1 💚	compile	4m 57s		master passed
+1 💚	checkstyle	1m 30s		master passed
+1 💚	spotbugs	4m 29s		master passed
+1 💚	spotless	0m 42s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 52s		the patch passed
+1 💚	compile	4m 58s		the patch passed
+1 💚	cc	4m 58s		the patch passed
+1 💚	javac	4m 58s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 30s		the patch passed
-0 ⚠️	rubocop	0m 10s	/results-rubocop.txt	The patch generated 12 new + 502 unchanged - 2 fixed = 514 total (was 504)
+1 💚	spotbugs	4m 50s		the patch passed
+1 💚	hadoopcheck	11m 8s		Patch does not cause any errors with Hadoop 3.3.6 3.4.1.
+1 💚	hbaseprotoc	1m 46s		the patch passed
+1 💚	spotless	0m 42s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		51m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7577/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7577
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless cc buflint bufcompat hbaseprotoc rubocop
uname	Linux 4b57a8ef0ca5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / cae2efb
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	85 (vs. ulimit of 30000)
modules	C: hbase-protocol-shaded hbase-client hbase-server hbase-shell U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7577/3/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3 rubocop=1.37.1
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

[Apache9]

Since we have a Configuration Object in ReplicationPeerConfig, what about just create a combined Configuration instance and use it when creating ReplicationSource? In this way you can directly change the sleepForRetry through the configuration.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 29s		Docker mode activated.
-0 ⚠️	yetus	0m 2s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	0m 21s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 23s		master passed
+1 💚	compile	2m 14s		master passed
+1 💚	javadoc	1m 5s		master passed
+1 💚	shadedjars	6m 15s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 10s		the patch passed
+1 💚	compile	2m 14s		the patch passed
+1 💚	javac	2m 14s		the patch passed
+1 💚	javadoc	1m 5s		the patch passed
+1 💚	shadedjars	6m 18s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	0m 34s		hbase-protocol-shaded in the patch passed.
+1 💚	unit	1m 32s		hbase-client in the patch passed.
+1 💚	unit	227m 33s		hbase-server in the patch passed.
+1 💚	unit	7m 19s		hbase-shell in the patch passed.
		268m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7577/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7577
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 6f712e9d0480 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / cae2efb
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7577/3/testReport/
Max. process+thread count	3729 (vs. ulimit of 30000)
modules	C: hbase-protocol-shaded hbase-client hbase-server hbase-shell U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7577/3/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

[sidkhillon]

Since we have a Configuration Object in ReplicationPeerConfig, what about just create a combined Configuration instance and use it when creating ReplicationSource? In this way you can directly change the sleepForRetry through the configuration.

Just to confirm, you're suggesting I use the existing configuration map in ReplicationPeerConfig and store the value as "replication.source.sleepforretries". Then, we can override the config by doing something like:

Configuration combinedConf = new Configuration(globalConf);
// Override any values in globalConf with the existing value in peerConfig
peerConfig.getConfiguration().forEach(combinedConf::set);
// set this.conf = combinedConf in the ReplicationSource

Set peer-specific override via update_peer_config '1', CONFIG => {"replication.source.sleepforretries" => 2000}

Is that the approach you are looking for?