[BUG] RFS - ElasticSearch 6.8 -> OpenSearch 2.x Local Testing Fails

by ADMIN 68 views

[BUG] RFS - ElasticSearch 6.8 -> OpenSearch 2.x local testing fails

What is the bug?

The bug is related to the migration process from ElasticSearch 6.8 to OpenSearch 2.x using the Reindex-From-Snapshot (RFS) process. The RFS process is designed to allow jumping versions, but in this case, the migration fails when trying to migrate from ES 6.8 to OS 2.x.

What are your migration environments?

The migration environments are set up using the following commands:

./gradlew DocumentsFromSnapshotMigration:es68ComposeDown && ./gradlew DocumentsFromSnapshotMigration:es68ComposeUp

The domains show:

  • ES 6.8.23
  • OS 2.11.1

How can one reproduce the bug?

To reproduce the bug, follow these steps:

  1. Create an S3 bucket
  2. Create and clean local /tmp dirs (s3 + lucene)
  3. Export AWS credentials with access to the S3 bucket
  4. Run the following commands:
./gradlew DocumentsFromSnapshotMigration:es68ComposeDown && ./gradlew DocumentsFromSnapshotMigration:es68ComposeUp
  1. Create a test document using curl:
curl "localhost:19200/test/_doc/1" --json '{"foo":true}'
  1. Create a snapshot using the following command:
./gradlew CreateSnapshot:run --args='--snapshot-name reindex-from-snapshot --s3-repo-uri s3://<redacted> --s3-region us-east-1 --source-host http://localhost:19200'
  1. Migrate metadata using the following command:
./gradlew MetadataMigration:run --args='migrate --source-version "ElasticSearch 6.8" --snapshot-name reindex-from-snapshot --s3-local-dir /tmp/s3_files --s3-repo-uri s3://<redacted> --s3-region us-east-1 --target-host http://localhost:29200'
  1. Backfill from snapshot using the following command:
./gradlew DocumentsFromSnapshotMigration:run --args='--source-version "ES 6.8" --snapshot-name reindex-from-snapshot --s3-local-dir /tmp/s3_files --s3-repo-uri s3://<redacted> --s3-region us-east-1 --lucene-dir /tmp/lucene_files --target-host http://localhost:29200'

What is the expected behavior?

The expected behavior is that the test index should be migrated to the OS 2.x domain. However, the data is not migrated, and the following logs are observed:

> Task :DocumentsFromSnapshotMigration:run
2025-03-12 12:19:09,865 INFO o.o.m.u.ProcessHelpers [main] getNodeInstanceName()=generated_50f2b502-9bf0-4431-b33e-d50b993d2c6d
Starting program with: --source-version ES 6.8 --snapshot-name reindex-from-snapshot --s3-local-dir /tmp/s3_files --s3-repo-uri s3://<redacted> --s3-region us-east-1 --lucene-dir /tmp/lucene_files --target-host http://localhost:29200
2025-03-12 12:19:09,867 INFO o.o.m.RfsMigrateDocuments [main] Starting RfsMigrateDocuments with workerId=generated_50f2b502-9bf0-4431-b33e-d50b993d2c6d
2025-03-12 12:19:09,888 WARN o.o.m.t.RootOtelContext [main] Collector endpoint=null, so serviceName parameter 'documentMigration' is being ignored since a no-op OpenTelemetry object is being created
2025-03-12 12:19:10,231 ERROR i.n.r.d.DnsServerAddressStreamProviders [main] Unable to load io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS. Check whether you have a dependency on 'io.netty:netty-resolver-dns-native-macos'. Use DEBUG level to see the full stack: java.lang.UnsatisfiedLinkError: failed to load the required native library
2025-03-12 12:19:10,357 INFO o.o.m.RfsMigrateDocuments [main] Doc Transformations config string: [  {    "JsonTransformerForDocumentTypeRemovalProvider":""  }]
2025-03-12 12:19:10,359 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transformation.JsonTransformerForDocumentTypeRemovalProvider@1686f0b4
2025-03-12 12:19:10,359 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.JsonConditionalTransformerProvider@61e3cf4d
2025-03-12 12:19:10,360 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.TypeMappingSanitizationTransformerProvider@1d61c6dc
2025-03-12 12:19:10,367 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.JsonJMESPathTransformerProvider@50a691d3
2025-03-12 12:19:10,368 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.JsonJinjavaTransformerProvider@87b5b49
2025-03-12 12:19:10,368 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.JsonJoltTransformerProvider@128c502c
2025-03-12 12:19:10,368 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.NoopTransformerProvider@3c4262d1
2025-03-12 12:19:10,369 INFO o.o.m.t.TransformationLoader [main] IJsonTransformerProviders loaded: class org.opensearch.migrations.transformation.JsonTransformerForDocumentTypeRemovalProvider; class org.opensearch.migrations.transform.JsonConditionalTransformerProvider; class org.opensearch.migrations.transform.TypeMappingSanitizationTransformerProvider; class org.opensearch.migrations.transform.JsonJMESPathTransformerProvider; class org.opensearch.migrations.transform.JsonJinjavaTransformerProvider; class org.opensearch.migrations.transform.JsonJoltTransformerProvider; class org.opensearch.migrations.transform.NoopTransformerProvider

2025-03-12 12:19:10,847 INFO o.o.m.c.ClusterProviderRegistry [main] Found snapshot resource reader for version: ELASTICSEARCH 6.8.0
2025-03-12 12:19:10,871 INFO o.o.m.b.w.OpenSearchWorkCoordinator [main] Creating .migrations_working_state because HEAD returned 404
2025-03-12 12:19:11,237 INFO o.o.m.b.w.ShardWorkPreparer [main] Acquired work to set the shard workitems
2025-03-12 12:19:11,237 INFO o.o.m.b.w.ShardWorkPreparer [main] Setting up the Documents Work Items...
2025-03-12 12:19:12,640 INFO o.o.m.b.w.ShardWorkPreparer [main] Index test has 5 shards
2025-03-12 12:19:12,640 INFO o.o.m.b.w.ShardWorkPreparer [main] Creating Documents Work Item for index: test, shard: 0
2025-03-12 12:19:12,651 INFO o.o.m.b.w.ShardWorkPreparer [main] Creating Documents Work Item for index: test, shard: 1
2025-03-12 12:19:12,658 INFO o.o.m.b.w.ShardWorkPreparer [main] Creating Documents Work Item for index: test, shard: 2
2025-03-12 12:19:12,665 INFO o.o.m.b.w.ShardWorkPreparer [main] Creating Documents Work Item for index: test, shard: 3
2025-03-12 12:19:12,671 INFO o.o.m.b.w.ShardWorkPreparer [main] Creating Documents Work Item for index: test, shard: 4
2025-03-12 12:19:12,678 INFO o.o.m.b.w.ShardWorkPreparer [main] Finished setting up the Documents Work Items.
2025-03-12 12:19:12,830 INFO o.o.m.b.w.OpenSearchWorkCoordinator [main] Returning work item and lease: OpenSearchWorkCoordinator.WorkItemWithPotentialSuccessors(workItemId=test__0__0, leaseExpirationTime=2025-03-12T12:29:12Z, successorWorkItemIds=[])
2025-03-12 12:19:12,830 INFO o.o.m.b.w.Documents<br/>
**[BUG] RFS - ElasticSearch 6.8 -> OpenSearch 2.x local testing fails**

### **Q&A**

**Q: What is the bug?**
A: The bug is related to the migration process from ElasticSearch 6.8 to OpenSearch 2.x using the Reindex-From-Snapshot (RFS) process. The RFS process is designed to allow jumping versions, but in this case, the migration fails when trying to migrate from ES 6.8 to OS 2.x.

**Q: What are the migration environments?**
A: The migration environments are set up using the following commands:

```bash
./gradlew DocumentsFromSnapshotMigration:es68ComposeDown && ./gradlew DocumentsFromSnapshotMigration:es68ComposeUp

The domains show:

  • ES 6.8.23
  • OS 2.11.1

Q: How can one reproduce the bug? A: To reproduce the bug, follow these steps:

  1. Create an S3 bucket
  2. Create and clean local /tmp dirs (s3 + lucene)
  3. Export AWS credentials with access to the S3 bucket
  4. Run the following commands:
./gradlew DocumentsFromSnapshotMigration:es68ComposeDown && ./gradlew DocumentsFromSnapshotMigration:es68ComposeUp
  1. Create a test document using curl:
curl "localhost:19200/test/_doc/1" --json '{"foo":true}'
  1. Create a snapshot using the following command:
./gradlew CreateSnapshot:run --args='--snapshot-name reindex-from-snapshot --s3-repo-uri s3://<redacted> --s3-region us-east-1 --source-host http://localhost:19200'
  1. Migrate metadata using the following command:
./gradlew MetadataMigration:run --args='migrate --source-version "ElasticSearch 6.8" --snapshot-name reindex-from-snapshot --s3-local-dir /tmp/s3_files --s3-repo-uri s3://<redacted> --s3-region us-east-1 --target-host http://localhost:29200'
  1. Backfill from snapshot using the following command:
./gradlew DocumentsFromSnapshotMigration:run --args='--source-version "ES 6.8" --snapshot-name reindex-from-snapshot --s3-local-dir /tmp/s3_files --s3-repo-uri s3://<redacted> --s3-region us-east-1 --lucene-dir /tmp/lucene_files --target-host http://localhost:29200'

Q: What is the expected behavior? A: The expected behavior is that the test index should be migrated to the OS 2.x domain. However, the data is not migrated, and the following logs are observed:

> Task :DocumentsFromSnapshotMigration:run
2025-03-12 12:19:09,865 INFO o.o.m.u.ProcessHelpers [main] getNodeInstanceName()=generated_50f2b502-9bf0-4431-b33e-d50b993d2c6d
Starting program with: --source-version ES 6.8 --snapshot-name reindex-from-snapshot --s3-local-dir /tmp/s3_files --s3-repo-uri s3://<redacted> --s3-region us-east-1 --lucene-dir /tmp/lucene_files --target-host http://localhost:29200
2025-03-12 12:19:09,867 INFO o.o.m.RfsMigrateDocuments [main] Starting RfsMigrateDocuments with workerId=generated_50f2b502-9bf0-4431-b33e-d50b993d2c6d
2025-03-12 12:19:09,888 WARN o.o.m.t.RootOtelContext [main] Collector endpoint=null, so serviceName parameter 'documentMigration' is being ignored since a no-op OpenTelemetry object is being created
2025-03-12 12:19:10,231 ERROR i.n.r.d.DnsServerAddressStreamProviders [main] Unable to load io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS. Check whether you have a dependency on 'io.netty:netty-resolver-dns-native-macos'. Use DEBUG level to see the full stack: java.lang.UnsatisfiedLinkError: failed to load the required native library
2025-03-12 12:19:10,357 INFO o.o.m.RfsMigrateDocuments [main] Doc Transformations config string: [  {    "JsonTransformerForDocumentTypeRemovalProvider":""  }]
2025-03-12 12:19:10,359 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transformation.JsonTransformerForDocumentTypeRemovalProvider@1686f0b4
2025-03-12 12:19:10,359 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.JsonConditionalTransformerProvider@61e3cf4d
2025-03-12 12:19:10,360 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.TypeMappingSanitizationTransformerProvider@1d61c6dc
2025-03-12 12:19:10,367 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.JsonJMESPathTransformerProvider@50a691d3
2025-03-12 12:19:10,368 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.JsonJinjavaTransformerProvider@87b5b49
2025-03-12 12:19:10,368 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.JsonJoltTransformerProvider@128c502c
2025-03-12 12:19:10,368 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.NoopTransformerProvider@3c4262d1
2025-03-12 12:19:10,369 INFO o.o.m.t.TransformationLoader [main] IJsonTransformerProviders loaded: class org.opensearch.migrations.transformation.JsonTransformerForDocumentTypeRemovalProvider; class org.opensearch.migrations.transform.JsonConditionalTransformerProvider; class org.opensearch.migrations.transform.TypeMappingSanitizationTransformerProvider; class org.opensearch.migrations.transform.JsonJMESPathTransformerProvider; class org.opensearch.migrations.transform.JsonJinjavaTransformerProvider; class org.opensearch.migrations.transform.JsonJoltTransformerProvider; class org.opensearch.migrations.transform.NoopTransformerProvider

2025-03-12 12:19:10,847 INFO o.o.m.c.ClusterProviderRegistry [main] Found snapshot resource reader for version: ELASTICSEARCH 6.8.0
2025-03-12 12:19:10,871 INFO o.o.m.b.w.OpenSearchWorkCoordinator [main] Creating .migrations_working_state because HEAD returned 404
2025-03-12 12:19:11,237 INFO o.o.m.b.w.ShardWorkPreparer [main] Acquired work to set the shard workitems
2025-03-12 12:19:11,237 INFO o.o.m.b.w.ShardWorkPreparer [main] Setting up the Documents Work Items...
2025-03-12 12:19:12,640 INFO o.o.m.b.w.ShardWorkPreparer [main] Index test has 5 shards
2025-03-12 12:19:12,640 INFO o.o.m.b.w.ShardWorkPreparer [main] Creating Documents Work Item for index: test, shard: 0
2025-03-12 12:19:12,651 INFO o.o.m.b.w.ShardWorkPreparer [main] Creating Documents Work Item for index: test, shard: 1
2025-03-12 12:19:12,658 INFO o.o.m.b.w.ShardWorkPreparer [main] Creating Documents Work Item for index: test, shard: 2
2025-03-12 12:19:12,665 INFO o.o.m.b.w.ShardWorkPreparer [main] Creating Documents Work Item for index: test, shard: 3
2025-03-12 12:19:12,671 INFO o.o.m.b.w.ShardWorkPreparer [main] Creating Documents Work Item for index: test, shard: 4
2025-03-12 12:19:12,678 INFO o.o.m.b.w.ShardWorkPreparer [main] Finished setting up the Documents Work Items.
2025-03-12 12:19:12,830 INFO o.o.m.b.w.OpenSearchWorkCoordinator [main] Returning work item and lease: OpenSearchWorkCoordinator.WorkItemWithPotentialSuccessors(workItemId=test__0__0, leaseExpirationTime=2025-03-12T12:29:12Z, successorWorkItemIds=[])
2025-03