Vertex with Id : <ID> was removed error when there is no drop or remove call which has happened.

[dxtr-1-0]

I am using JanusGraph as a base to power a graph service in my org. One day randomly for few queries I started getting an error like following :
Caused by: java.lang.IllegalStateException: Vertex with id <ID> was removed. org.janusgraph.core.InvalidElementException.removedException(InvalidElementException.java:57) org.janusgraph.graphdb.vertices.AbstractVertex.it(AbstractVertex.java:60) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.query(StandardJanusGraphTx.java:1226) at org.janusgraph.graphdb.vertices.AbstractVertex.getVertexLabelInternal(AbstractVertex.java:126) at org.janusgraph.graphdb.vertices.AbstractVertex.vertexLabel(AbstractVertex.java:131) at org.janusgraph.graphdb.vertices.AbstractVertex.label(AbstractVertex.java:122) at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV3$EdgeJacksonSerializer.serialize(GraphSONSerializersV3.java:167) at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV3$EdgeJacksonSerializer.serialize(GraphSONSerializersV3.java:145)

I am using custom id approach to generate the vertex ids, using hbase as the backend data store. I checked in hbase and for the row key that is constructed out of the vertex id for which i get this error, i am able to get some data from hbase shell.
On doing query for this vertex like
g.V().has('some_identifier_property', 'property value')
I am getting empty result,
and similarly for query like
g.V('CUSTOM_ID').hasNext() \\OR g.V('CUSTOM_ID').next()
I am getting empty result. For queries where i am fetching edges where this vertex is at the other end, the call breaks with the error shown above.
I am a bit clueless on how to proceed further in debugging and this is causing us to re-evaluate if JanusGraph can be inconsistent in some scenarios. Any help would be appreciated as this is super critical for functioning of my project.
@li-boxuan @mbroecheler @farodin91 saw your commits in the hbaseStoreManager so tagging you for reference.

[li-boxuan]

Did you by chance use the reserved character in your vertex ID? https://docs.janusgraph.org/advanced-topics/custom-vertex-id/#override-reserved-character

How frequent does this happen? How common is it?

i am able to get some data from hbase shell

This might still be a ghost vertex. Have you checked out https://docs.janusgraph.org/advanced-topics/eventual-consistency/#ghost-vertices?

[dxtr-1-0]

Which column family should this column belong to? Also, how i construct the row key for the custom id is basically by adding \x80 as the prefix and \xB before the last character of the custom id. Against this rowkey only column family 'e' is present so i am guessing there is some other id or ids which janusgraph generates and stores other cfs against those ids. Please let me know which row key and which cf should i search in for this column.

[dxtr-1-0]

I am guessing while updating a vertex with new properties or any other similar operation, the lifecycle is being set to removed temporarily but this change is not being reverted upon some failure scenario.

[dxtr-1-0]

I found out an issue in edge addition path.
g.V(sourceId).as("a") .V(destinationId).as("b") .addE(edgeLabel) .from("a").to("b");

adding an edge like this, if the vertex "b" does not exist, janusgraph still creates an edge between "a" and (i am guessing) a cached vertex "b" but on querying something like give me all edges from "a" , the query breaks with the above error in the issue. Reason being janusgraph not being able to find the vertex "b" while traversing edge between "a" and "b". Although the error propagated is misleading, as it says vertex "b" was removed when in reality it was never even created in the first place.

In my opinion the expected behaviour should be: Either dont allow creation of edge between 2 nodes if both of them don't exist or persist the auto created nodes in its original form, currently the auto created node info is persisted but in removed state in the DB.

[li-boxuan]

@dxtr-1-0 have you tried out checkInternalVertexExistence option in https://docs.janusgraph.org/basics/transactions/#transaction-configuration? That might solve the issue

[dxtr-1-0]

@li-boxuan Yes I have tried that but it slows down performance by a good amount. For now i have implemented a hard check on the application layer. It also impacts performance but for some reason not as much as checkInternalVertexExistence. Any clue why this happens? For reference, without any check my p99 latencies were 90-100ms. With application level checks my latencies are 150-200ms p99. And with checkInternalVertexExistence, my p99 latencies were 700-750ms. This difference was not acceptable according to the SLOs of my service, hence i had to resort to application level checks but i am not entirely sure if the application level checks are foolproof to the full extent or not..