Skip to content

ATLAS-5220 : Add Maven profiles to build slim server by selecting only required storage/index backend - #658

Open
VinayakMarraiya230920 wants to merge 9 commits into
apache:masterfrom
VinayakMarraiya230920:ATLAS-5220
Open

ATLAS-5220 : Add Maven profiles to build slim server by selecting only required storage/index backend#658
VinayakMarraiya230920 wants to merge 9 commits into
apache:masterfrom
VinayakMarraiya230920:ATLAS-5220

Conversation

@VinayakMarraiya230920

@VinayakMarraiya230920 VinayakMarraiya230920 commented Jun 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

ATLAS-5220: Add Maven profiles for slim server packaging

Atlas server builds can now include only the graph storage, search index, and audit backends needed for a deployment, instead of bundling every JanusGraph and audit JAR on every build.

New Maven profiles (use with -Pdist):

Storage: storage-hbase (default), storage-cassandra, storage-berkeleyje, storage-rdbms
Index: index-solr (default), index-elasticsearch
Audit: audit-hbase (default), audit-inmemory, audit-rdbms, audit-cassandra, audit-noop
Local dev: embedded-hbase-solr (bundles HBase + Solr in the tarball)
Main changes:

Split audit into separate modules (audit-hbase, audit-rdbms, audit-cassandra) and profile-gate WAR dependencies
Filter atlas-application.properties and atlas-env.sh to match selected backends
Fail fast at startup if runtime config does not match the build profile
Updated build docs in README.md and BuildInstruction.md
Example:

mvn clean package -DskipTests -Pdist,storage-rdbms,index-solr,audit-rdbms

How was this patch tested?

Unit tests:

mvn test -pl graphdb/janus -Dtest=AtlasBuildBackendAlignmentTest -DskipDocs -Drat.skip=true
Result: 2 tests passed.

Manual build verification (JDK 1.8): Built with each profile combination and checked:

WAR contains only the expected backend JARs
atlas-application.properties has the correct EntityAuditRepository and storage/index settings
Embedded build produces distro/target/hbase, distro/target/solr, and sets MANAGE_LOCAL_HBASE/SOLR=true
mvn package does not start Atlas (port 21000 not listening)
All 11 profile/packaging scenarios passed. No server startup required for these tests.

@VinayakMarraiya230920
VinayakMarraiya230920 force-pushed the ATLAS-5220 branch 3 times, most recently from b376fca to 6d346b8 Compare June 22, 2026 13:07
pinal-shah
pinal-shah reviewed Jun 23, 2026
View reviewed changes
Comment thread graphdb/janus/pom.xml Outdated

@pinal-shah pinal-shah Jun 23, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please review, if this can also be added as profile based

@VinayakMarraiya230920 VinayakMarraiya230920 Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

pinal-shah
pinal-shah reviewed Jun 23, 2026
View reviewed changes
Comment thread graphdb/janusgraph-rdbms/pom.xml
@@ -40,7 +40,8 @@
</dependency>
<dependency>
<groupId>org.apache.atlas</groupId>
<artifactId>atlas-intg</artifactId>
<artifactId>atlas-graphdb-api</artifactId>

@pinal-shah pinal-shah Jun 23, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please review if this change is needed

@VinayakMarraiya230920 VinayakMarraiya230920 Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, needed — module uses AtlasUniqueKeyHandler from atlas-graphdb-api; ApplicationProperties/AtlasException come transitively.

pinal-shah
pinal-shah reviewed Jun 23, 2026
View reviewed changes
Comment thread distro/pom.xml
For a local, ready-to-run tarball with bundled HBase+Solr and dev-oriented defaults, use -Pembedded-hbase-solr.
Other examples: Berkeley/RDBMS/Cassandra blocks below; Berkeley+Solr bundle: -Pberkeley-solr.
-->
<profile>

@pinal-shah pinal-shah Jun 23, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add READMe for atlas build

@VinayakMarraiya230920 VinayakMarraiya230920 Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added slim build docs to README.md

@VinayakMarraiya230920
VinayakMarraiya230920 force-pushed the ATLAS-5220 branch 2 times, most recently from d2046ac to cda6d55 Compare August 17, 2026 07:32
@VinayakMarraiya230920
VinayakMarraiya230920 marked this pull request as ready for review September 30, 2026 06:17
@VinayakMarraiya230920
ATLAS-5220: Add Maven profiles to build slim server by selecting only…
20c4c8e 
… required storage/index backend

Introduce storage-* and index-* Maven profiles so a slim Atlas server can be
built with only the required JanusGraph storage/index backends. Default
packaging uses the HBase storage backend and the Solr index backend; optional
BerkeleyDB/Cassandra/RDBMS storage and Elasticsearch/Lucene index backends are
profile-scoped. Adds the hbase-shaded-client-fixed module and the related
build, distro, and configuration updates.
@VinayakMarraiya230920
ATLAS-5220: Resolve docker build failure on PR CI
5026299 
- Skip SpotBugs for hbase-shaded-client-fixed module: it only unpacks and
  repacks the third-party hbase-shaded-client jar into target/classes, so
  SpotBugs analyzed the entire shaded client and failed the findbugs-check
  step (OutOfMemoryError). Set skipCheck=true for this module.
- Pin graphdb/janus unit tests to the embedded berkeleyje storage backend
  (test-scoped janusgraph-berkeleyje + graph.storage.backend override) so
  they run offline instead of timing out against HBase/Zookeeper. Production
  packaging still defaults to hbase2.
@VinayakMarraiya230920
ATLAS-5220: Fix ClassNotFoundException for HBaseBasedAuditRepository …
d787dab 
…in tests under -Pdist

The audit backends were split into separate atlas-audit-* modules, moving
HBaseBasedAuditRepository (the production default and the -Pdist value of
${entity.repository.impl}) out of atlas-repository into atlas-audit-hbase.
atlas-audit-hbase depends on atlas-repository, so it cannot be added as a test
dependency of intg/repository without creating a dependency cycle.

Under the CI build (mvn -Pdist,embedded-solr-it ... verify) the filtered test
property atlas.EntityAuditRepository.impl resolved to HBaseBasedAuditRepository,
which is absent from the intg/repository test classpath, so building the Guice
injector failed with:
  java.lang.ClassNotFoundException:
  org.apache.atlas.repository.audit.HBaseBasedAuditRepository

Hard-code the in-memory audit repository (bundled in atlas-repository) in the
intg and authorization test atlas-application.properties, matching what webapp
already does. Tests become backend-agnostic and no longer depend on the
packaging profile; the distro's shipped config still uses
${entity.repository.properties} and is unchanged.
@VinayakMarraiya230920
ATLAS-5220: use embedded berkeleyje backend for offline module tests
4e6caf7 
ATLAS-5220 made the JanusGraph storage backends profile-scoped and changed the
test-facing graph backend default to hbase2. Under the CI build (-Pdist), the
atlas-repository and atlas-webapp tests open a real JanusGraph and failed offline
with "JanusGraphException: Could not open global configuration" because hbase2
needs a live HBase/Zookeeper.

Fix (test-only, no impact on distro packaging):
- intg test atlas-application.properties: hard-code atlas.graph.storage.backend=berkeleyje
  (consumed via the intg test-jar by atlas-repository tests).
- repository/pom.xml, webapp/pom.xml: add janusgraph-berkeleyje as a test-scoped
  dependency so the embedded backend is on the test classpath under -Pdist.

Production and distribution continue to use the configured backend (hbase2) and are
unaffected. Compatible with JDK 8 and JDK 17.
@VinayakMarraiya230920
ATLAS-5220: make TaskExecutor.waitUntilDone() deterministic to fix fl…
a6e66ff 
…aky CI on JDK 8

The CI docker-build matrix (java: [8, 17]) intermittently failed on JDK 8 in
TaskExecutorTest.persistedIsExecuted with "expected [1] but found [0]" while
asserting the task statistics (getTotalSuccess()/getTotalError()).

Root cause: TaskExecutor.waitUntilDone() (a @VisibleForTesting helper used only
by the tests) waited with a fixed Thread.sleep(5000). The tasks run on a
single-threaded executor and perform real JanusGraph (berkeleyje) commits, so on
slower JDK 8 CI runners they occasionally had not finished - and the statistics
counters had not been updated - when the assertions ran. JDK 17 was faster and
usually finished in time; because the matrix is fail-fast, the JDK 8 failure also
cancelled the JDK 17 job.

Fix (test-only, no production runtime path changes): replace the fixed sleep with
a barrier. The executor is single-threaded and FIFO, so submitting a no-op task
and blocking on its Future guarantees every previously submitted task has
finished (and its statistics recorded) before the assertions run. This is
deterministic regardless of host/JVM speed and removes the flakiness on both
JDK 8 and JDK 17.

Verified with: mvn -pl repository test -Pdist -Dtest=TaskExecutorTest (JDK 8) -> 3/3 passed.
@VinayakMarraiya230920
ATLAS-5220: make TaskManagementTest deterministic to fix flaky CI on …
aa091c0 
…JDK 8

The CI docker-build matrix (java: [8, 17]) intermittently failed on JDK 8 in
TaskManagementTest.taskSucceedsTaskVertexRemoved with a NullPointerException at
the assertion asserting the spied tasks were performed.

Root cause: the test submitted the tasks via TaskManagement.addAll(...) and then
waited with a fixed TimeUnit.SECONDS.sleep(5) before asserting. The tasks run on
a single-threaded executor and perform real JanusGraph (berkeleyje) commits, so
on slower JDK 8 CI runners the error task had not been created/executed yet -
SpyingFactory.getErrorTask() was still null - and getErrorTask().taskPerformed()
threw an NPE. JDK 17 was faster and usually finished within the 5s window;
because the matrix is fail-fast, the JDK 8 failure also cancelled the JDK 17 job.

Fix (test-only, no production changes): replace the fixed sleep with a bounded
poll that waits (up to 30s) until both spied tasks have actually executed and the
completed add-task vertex has been removed, then asserts. This is deterministic
regardless of host/JVM speed and removes the flakiness on both JDK 8 and JDK 17.

Verified with: mvn -pl repository test -Pdist
-Dtest=TaskManagementTest,TaskExecutorTest,TaskRegistryTest (JDK 8) -> 8/8 passed.
@VinayakMarraiya230920
ATLAS-5220: fix checkstyle violation in TaskExecutor empty lambda body
dbd7fb2 
The dist build runs checkstyle (-Pdist sets checkstyle.skip=false,
failOnViolation=true) with ./dev-support/checkstyle.xml, which has a
RegexpMultiline rule "Whitespace inside empty lambda body" (format: ->\s*\{\s+\}).

The barrier added in waitUntilDone() used "() -> { }" (whitespace between the
braces), which matched that rule and failed the atlas-repository build:
  TaskExecutor.java:[75] (regexp) RegexpMultiline: Whitespace inside empty lambda body

Remove the whitespace: "() -> {}". WhitespaceAround allows empty lambdas
(allowEmptyLambdas=true), so the no-space form is compliant.

Verified with: mvn -pl repository checkstyle:check with the dist ruleset -> BUILD SUCCESS.
@VinayakMarraiya230920
ATLAS-5220: make embedded berkeleyje backend available to webapp inte…
0d0fd3a 
…gration tests

The profile-slim WAR (storage-hbase by default) no longer bundles the
janusgraph-berkeleyje backend, and the jetty deploy-war goal does not add
test-scoped dependency jars to the server classpath. As a result the embedded
integration-test server could not load BerkeleyJEStoreManager, never reached
ACTIVE state, and every REST call returned HTTP 503.

Copy janusgraph-berkeleyje (and its je dependency) into target/it-backend-libs
via maven-dependency-plugin and add them to the jetty extraClasspath so the IT
server can initialise JanusGraph on berkeleyje, while keeping the shipped WAR slim.

resolves docker build issue on PR
@VinayakMarraiya230920
ATLAS-5220: resolves docker build issue - add berkeleyje IT backend t…
4a11b0d 
…o addon bridge embedded server classpath

The slim default WAR (storage-hbase) no longer bundles janusgraph-berkeleyje,
so the embedded Atlas server started by each addon bridge's jetty deploy-war
for integration tests could not load BerkeleyJEStoreManager and never reached
ACTIVE, causing 503 responses on all addon ITs.

Add the berkeleyje backend (and its je dependency, copied by the webapp
copy-it-backend-libs execution) to the IT server classpath via jetty
<extraClasspath> in the hive, falcon, hbase, impala, kafka, sqoop and storm
bridge poms. The shipped WAR stays slim; this only affects the test-time
embedded server.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pinal-shah pinal-shah pinal-shah left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@VinayakMarraiya230920 @pinal-shah