HBASE-30473 Expose Master switch states as numeric metrics by junegunn · Pull Request #8758 · apache/hbase

junegunn · 2026-10-12T07:26:22Z

Jira: https://issues.apache.org/jira/browse/HBASE-30473

Rendered doc

Metric

{
   "name": "Hadoop:service=HBase,name=Master,sub=Server",
   "modelerType": "Master,sub=Server",
   "tag.liveRegionServers": "192.168.35.231,16020,1791788928190",
   "tag.deadRegionServers": "",
   "tag.drainingRegionServers": "",
   "tag.zookeeperQuorum": "127.0.0.1:2181",
   "tag.serverName": "192.168.35.231,16000,1791788926592",
   "tag.clusterId": "07aeab61-7d0f-487b-9f71-4bf8d38565c1",
   "tag.isActiveMaster": "true",
   "tag.Context": "master",
   "tag.Hostname": "AD02514045",
   "mergePlanCount": 0,
   "splitPlanCount": 0,
   "masterActiveTime": 1791788928555,
   "masterStartTime": 1791788926592,
   "masterFinishedInitializationTime": 1791788934045,
   "averageLoad": 6.0,
   "onlineRegionCount": 5,
   "offlineRegionCount": 0,
   "numRegionServers": 1,
   "numDeadRegionServers": 0,
   "numDrainingRegionServers": 0,
   "oldWALsDirSize": 0,
   "catalogJanitorEnabled": 1,
   "mergeEnabled": 1,
   "normalizerEnabled": 1,
   "snapshotCleanupEnabled": 1,
   "splitEnabled": 1,
   "replicationPeerModificationEnabled": 1,
   "cleanerChoreEnabled": 1,
   "rpcThrottleEnabled": 0,
   "balancerEnabled": 1,
   "clusterRequests": 19,
   "clusterWriteRequests": 10,
   "clusterReadRequests": 18,
   "ServerCrashTime_num_ops": 1,
   "ServerCrashTime_min": 0,
   "ServerCrashTime_max": 0,
   "ServerCrashTime_mean": 0,
   "ServerCrashTime_25th_percentile": 0,
   "ServerCrashTime_median": 0,
   "ServerCrashTime_75th_percentile": 0,
   "ServerCrashTime_90th_percentile": 0,
   "ServerCrashTime_95th_percentile": 0,
   "ServerCrashTime_98th_percentile": 0,
   "ServerCrashTime_99th_percentile": 0,
   "ServerCrashTime_99.9th_percentile": 0,
   "ServerCrashSubmittedCount": 1,
   "ServerCrashFailedCount": 0
}

- Detect switches left disabled after maintenance. - Avoid quota-table reads during metrics collection. - Distinguish unavailable quota states from disabled states.

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open findings

The table rows start with || which will render as an extra empty first column in Markdown/MDX… · New
getSwitchStates() allocates a new Map on every metrics collection. Since metrics can be scraped… · New

🧠 Review effort: Lite

NihalJain · 2026-10-12T16:50:19Z

+  @Override
+  public Map<String, Integer> getSwitchStates() {
+    if (!master.isActiveMaster() || !master.isInitialized()) {
+      return Collections.emptyMap();


Agree on omitting these on backup masters, but for an active-but-not-yet-initialized master, would it be better to emit -1 for all keys? That matches the documented “unknown” semantics and keeps the series continuous across failover?

Thanks for review. You're right that it's more consistent.

keeps the series continuous across failover?

I think emitting -1 during master restart, which can be quite frequent in some cases, could make monitoring more complicated without giving much benefit.

So on second thought, how about removing -1 state altogether and also omitting rpcThrottleEnabled if it's not ready?

I've prepared the change to remove -1 and omit unavailable states, keeping the metrics strictly boolean. Please take a look and let me know what you think.

5d9fd6c

NihalJain · 2026-10-12T17:06:44Z

+  @Test
+  public void testEachSwitchChangesItsGauge() {
+    Map<String, Consumer<Boolean>> switches = new HashMap<>();
+    switches.put("balancerEnabled", enabled -> when(master.isBalancerOn()).thenReturn(enabled));


nit: Use MetricsMasterSource.*_NAME constants instead?

Good point, addressed in cd37bb7.

Oops, b0d816b as well.

NihalJain · 2026-10-12T17:10:32Z

+      return Collections.emptyMap();
+    }
+    int rpcThrottleEnabled = 0;
+    if (QuotaUtil.isQuotaEnabled(master.getConfiguration())) {


In maintenance mode finishActiveMasterInitialization returns before initQuotaManager(), so with quota enabled rpcThrottleEnabled stays -1 for the lifetime of the Master. The docs say “until the quota manager is ready”, which implies it’s transient. Could you document this case of maintenance, or report 0 like the other switches already do in maintenance mode?

Worth testing out the metrics in MAINTENANCE_MODE = "hbase.master.maintenance_mode"

5d9fd6c makes the metrics strictly boolean.

NihalJain

Overall LGTM, just added few nit comments

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open findings

The backslash escapes (\\|) are inside inline code spans, where the backslash typically renders… · New
The test hard-codes metric names as string literals. To reduce brittleness and ensure refactors… · New
verifyNoInteractions(master.getMasterQuotaManager()) calls getMasterQuotaManager() as part of… · New

2 resolved since last review

🧠 Review effort: Lite

Keep switch values boolean during initialization and maintenance mode.

NihalJain

+1, LGTM!

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open findings

This directly dereferences master.snapshotCleanupStateStore and will throw if it is ever… · New
The test hard-codes metric names as string literals even though this PR introduces… · New
assertEquals(8, states.size()) is brittle: it will fail if new switch-state gauges are added in… · New
The Markdown table uses a double leading pipe (||), which typically renders as an extra empty… · New

3 resolved since last review

🧠 Review effort: Lite

+    states.put(MetricsMasterSource.SNAPSHOT_CLEANUP_ENABLED_NAME,
+      master.snapshotCleanupStateStore.get() ? 1 : 0);


+    Map<String, Integer> states = new MetricsMasterWrapperImpl(master).getSwitchStates();
+    assertEquals(8, states.size());
+    assertFalse(states.containsKey(MetricsMasterSource.RPC_THROTTLE_ENABLED_NAME));


junegunn requested a balanced review from Copilot October 12, 2026 07:26

junegunn self-assigned this Oct 12, 2026

This comment was marked as off-topic.

Sign in to view

junegunn requested a lite review from Copilot October 12, 2026 07:29

HBASE-30473 Expose Master switch states as numeric metrics

c00f845

- Detect switches left disabled after maintenance. - Avoid quota-table reads during metrics collection. - Distinguish unavailable quota states from disabled states.

junegunn force-pushed the HBASE-30473 branch from 9cdf295 to c00f845 Compare October 12, 2026 07:35

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Comment thread ...ite/app/pages/_docs/docs/_mdx/(multi-page)/operational-management/metrics-and-monitoring.mdx Outdated

Comment thread hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterWrapperImpl.java Outdated

Copilot stopped reviewing on behalf of junegunn due to an error October 12, 2026 08:03
An unexpected error occurred. For more details, see the detailed logs in GitHub Actions.

NihalJain reviewed Oct 12, 2026

View reviewed changes

Comment thread hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterMetrics.java Outdated

NihalJain reviewed Oct 12, 2026

View reviewed changes

NihalJain requested a balanced review from Copilot October 12, 2026 17:12

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Copilot stopped reviewing on behalf of NihalJain due to an error October 12, 2026 18:19
The job was not acquired by a runner even after multiple attempts.

junegunn added 3 commits October 13, 2026 10:25

HBASE-30473 Use metric constants in switch unit tests

cd37bb7

HBASE-30473 Explain disabled RPC throttle in metrics test

6d5fe40

HBASE-30473 Omit unavailable RPC throttle state

5d9fd6c

Keep switch values boolean during initialization and maintenance mode.

NihalJain approved these changes Oct 13, 2026

View reviewed changes

NihalJain requested a balanced review from Copilot October 13, 2026 10:39

Copilot AI reviewed Oct 13, 2026

View reviewed changes

Copilot stopped reviewing on behalf of NihalJain due to an error October 13, 2026 14:52
The job was not acquired by a runner even after multiple attempts.

HBASE-30473 Use metric constants in TestMasterMetrics

b0d816b

		states.put(MetricsMasterSource.SNAPSHOT_CLEANUP_ENABLED_NAME,
		master.snapshotCleanupStateStore.get() ? 1 : 0);

Conversation

junegunn commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rendered doc

Metric

Uh oh!

This comment was marked as off-topic.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

NihalJain Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

junegunn Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

junegunn Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

NihalJain Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

junegunn Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

junegunn Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

NihalJain Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

junegunn Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

NihalJain left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

NihalJain left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

junegunn commented Oct 12, 2026 •

edited

Loading

NihalJain Oct 12, 2026 •

edited

Loading

junegunn Oct 13, 2026 •

edited

Loading

junegunn Oct 13, 2026 •

edited

Loading