---
title: "Advanced Features: Vector Search, Temporal Queries, and GraphRAG"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Advanced Features: Vector Search, Temporal Queries, and GraphRAG}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  eval = FALSE
)
```

This vignette covers the advanced capabilities of the AstraeaDB R package:
vector similarity search, hybrid and semantic search, temporal (time-travel)
queries, GraphRAG for LLM integration, Apache Arrow Flight transport, and
authentication.

All examples assume you have a running AstraeaDB server and a connected
client:

```{r connect}
library(AstraeaDB)
client <- astraea_connect()
```

## Vector Search

### Creating Nodes with Embeddings

Any node can carry a dense embedding vector. Pass the `embedding` argument
when creating the node:

```{r create-embeddings}
# Create nodes representing documents, each with a 4-dimensional embedding
doc1 <- client$create_node(
  labels     = c("Document"),
  properties = list(title = "Graph Databases", topic = "databases"),
  embedding  = c(0.9, 0.1, 0.2, 0.05)
)

doc2 <- client$create_node(
  labels     = c("Document"),
  properties = list(title = "Vector Search", topic = "search"),
  embedding  = c(0.1, 0.85, 0.3, 0.1)
)

doc3 <- client$create_node(
  labels     = c("Document"),
  properties = list(title = "Neural Networks", topic = "ML"),
  embedding  = c(0.15, 0.8, 0.9, 0.7)
)

doc4 <- client$create_node(
  labels     = c("Document"),
  properties = list(title = "Knowledge Graphs", topic = "databases"),
  embedding  = c(0.85, 0.2, 0.25, 0.1)
)
```

Embedding dimensionality is flexible -- AstraeaDB does not enforce a fixed
size, though all embeddings used together in a search should share the same
dimensionality.

### k-Nearest Neighbor Search

Use `vector_search()` to find the `k` nodes whose embeddings are most
similar to a query vector:

```{r vector-search}
results <- client$vector_search(
  query_vector = c(0.88, 0.15, 0.2, 0.08),
  k            = 3L
)

for (r in results) {
  cat(sprintf("Node %d -- distance: %.4f\n", r$node_id, r$distance))
}
#> Node 1 -- distance: 0.0048
#> Node 4 -- distance: 0.0129
#> Node 2 -- distance: 0.5877
```

Each result entry contains at least `node_id` and `distance` (smaller is
closer). A legacy `score` field equal to `distance` is also returned for
backward compatibility. The distance metric is determined by the server
configuration (typically cosine distance).

## Hybrid Search

Hybrid search combines **graph proximity** (how close a node is to an anchor
node in the graph) with **vector similarity** (how close its embedding is to
a query vector). The `alpha` parameter controls the blend:

- `alpha = 0.0` -- Pure graph proximity (closer in hops = higher rank).
- `alpha = 1.0` -- Pure vector similarity (ignore graph structure).
- `alpha = 0.5` -- Equal weight to both signals.

```{r hybrid-search}
# Find nodes that are both near doc1 in the graph AND similar to a query vector
results <- client$hybrid_search(
  anchor       = doc1,
  query_vector = c(0.8, 0.3, 0.25, 0.1),
  max_hops     = 3L,
  k            = 5L,
  alpha        = 0.7
)

for (r in results) {
  cat(sprintf("Node %d -- combined score: %.4f\n", r$node_id, r$score))
}
```

**Use case:** In a knowledge base, you want to find documents that are
topically similar to a query (vector similarity) and also structurally
related to a known starting point (graph proximity). By tuning `alpha` you
can control how much the graph structure influences the ranking.

## Semantic Search

Semantic search operations use a "concept" embedding to rank or navigate
the graph. Unlike vector search, which operates globally, semantic search
is local -- it starts from a specific node and considers only its
neighborhood.

### Semantic Neighbors

`semantic_neighbors()` retrieves the neighbors of a node and ranks them by
how similar their embeddings are to a concept vector:

```{r semantic-neighbors}
# First, connect the documents with edges
client$create_edge(doc1, doc2, "RELATED_TO")
client$create_edge(doc1, doc3, "RELATED_TO")
client$create_edge(doc1, doc4, "RELATED_TO")

# Rank doc1's neighbors by similarity to a "search-oriented" concept
ranked <- client$semantic_neighbors(
  node_id   = doc1,
  concept   = c(0.1, 0.9, 0.3, 0.1),
  direction = "outgoing",
  k         = 5L
)

for (r in ranked) {
  cat(sprintf("Neighbor %d -- distance: %.4f\n", r$node_id, r$distance))
}
```

### Semantic Walk

`semantic_walk()` performs a greedy traversal of the graph. At each step it
moves to the neighbor whose embedding is most similar to the concept vector:

```{r semantic-walk}
# Walk the graph following the "ML" concept
path <- client$semantic_walk(
  start    = doc1,
  concept  = c(0.1, 0.8, 0.9, 0.7),
  max_hops = 4L
)

cat("Walk path:", paste(path, collapse = " -> "), "\n")
```

This is useful for exploring how a topic "flows" through a knowledge graph.

## Temporal Queries (Time-Travel)

AstraeaDB edges can have temporal validity windows. When an edge has
`valid_from` and/or `valid_to` set (milliseconds since the Unix epoch), it
is only visible in temporal queries whose timestamp falls within the window.

### Setting Up Temporal Edges

```{r temporal-setup}
# Create a social network with temporal edges
alice <- client$create_node(c("Person"), list(name = "Alice"))
bob   <- client$create_node(c("Person"), list(name = "Bob"))
carol <- client$create_node(c("Person"), list(name = "Carol"))
dave  <- client$create_node(c("Person"), list(name = "Dave"))

# Helper: convert date string to epoch milliseconds
to_epoch_ms <- function(date_str) {
  as.numeric(as.POSIXct(date_str, tz = "UTC")) * 1000
}

# Alice knew Bob from 2020 to 2022
client$create_edge(
  source     = alice,
  target     = bob,
  edge_type  = "FRIENDS",
  valid_from = to_epoch_ms("2020-01-01"),
  valid_to   = to_epoch_ms("2022-01-01")
)

# Alice has known Carol since 2021 (no end date -- still active)
client$create_edge(
  source     = alice,
  target     = carol,
  edge_type  = "FRIENDS",
  valid_from = to_epoch_ms("2021-06-01")
)

# Alice has known Dave since 2023
client$create_edge(
  source     = alice,
  target     = dave,
  edge_type  = "FRIENDS",
  valid_from = to_epoch_ms("2023-01-01")
)
```

### Querying at a Point in Time

#### neighbors_at

```{r neighbors-at}
# Who was Alice friends with on July 1, 2021?
mid_2021 <- to_epoch_ms("2021-07-01")
friends_2021 <- client$neighbors_at(alice, "outgoing", mid_2021)

# Result includes Bob and Carol, but not Dave (not yet friends)
for (f in friends_2021) {
  node <- client$get_node(f$node_id)
  cat(node$properties$name, "\n")
}
#> Bob
#> Carol
```

#### bfs_at

```{r bfs-at}
# BFS at a point in time
bfs_2021 <- client$bfs_at(alice, max_depth = 2L, timestamp = mid_2021)

for (entry in bfs_2021) {
  cat(sprintf("Node %d at depth %d\n", entry$node_id, entry$depth))
}
```

#### shortest_path_at

```{r shortest-path-at}
# Shortest path at a specific time
early_2024 <- to_epoch_ms("2024-01-15")

sp <- client$shortest_path_at(
  from_node = alice,
  to_node   = dave,
  timestamp = early_2024,
  weighted  = FALSE
)

cat("Path:", paste(sp$path, collapse = " -> "), "\n")
cat("Hops:", sp$length, "\n")
```

### Use Case: Evolving Relationships

Temporal queries allow you to answer questions such as:

- "Who were Alice's collaborators in Q3 2022?"
- "What did the team structure look like before the reorg?"
- "How has the shortest communication path between two people changed over
  time?"

By sweeping a timestamp across a range, you can reconstruct the evolution
of a graph.

## GraphRAG

GraphRAG integrates graph data with large language models. AstraeaDB provides
two methods:

1. **`extract_subgraph()`** -- Extracts a subgraph centered on a node and
   linearizes it to text.
2. **`graph_rag()`** -- Full pipeline that extracts a subgraph, linearizes
   it, and sends the context and a question to a language model.

### Subgraph Extraction

```{r extract-subgraph}
# Extract a 2-hop subgraph around Alice, linearized as structured text
sg <- client$extract_subgraph(
  center    = alice,
  hops      = 2L,
  max_nodes = 20L,
  format    = "structured"
)

cat("Nodes:", sg$node_count, "\n")
cat("Edges:", sg$edge_count, "\n")
cat("\n", sg$text, "\n")
```

The `format` parameter controls how the subgraph is serialized:

| Format         | Description                                                |
|:---------------|:-----------------------------------------------------------|
| `"structured"` | Indented, human-readable summary of nodes and edges.       |
| `"prose"`      | Natural-language narrative describing the subgraph.        |
| `"triples"`    | List of `(subject, predicate, object)` triples.            |
| `"json"`       | Machine-readable JSON representation.                      |

### Full GraphRAG Pipeline

`graph_rag()` sends the extracted subgraph context along with your question
to the language model configured on the server:

```{r graph-rag}
answer <- client$graph_rag(
  question  = "Who are Alice's current friends and what do they work on?",
  anchor    = alice,
  hops      = 2L,
  max_nodes = 30L,
  format    = "prose"
)

cat(answer$answer, "\n")
```

You can also provide a `question_embedding` instead of (or in addition to) an
`anchor` node. This lets the server find the most relevant anchor automatically
via vector search:

```{r graph-rag-embedding}
answer <- client$graph_rag(
  question            = "What research topics are related to graph databases?",
  question_embedding  = c(0.9, 0.1, 0.2, 0.05),
  hops                = 3L,
  max_nodes           = 50L,
  format              = "structured"
)

cat(answer$answer, "\n")
```

## Graph Algorithms

AstraeaDB computes classic graph-analytics algorithms server-side and returns
the results over the wire. Pass `nodes =` to any of them to restrict the
computation to a subset of node IDs; the default is the whole graph.

### Centrality and Ranking

```{r algorithms-centrality}
# PageRank
scores <- client$run_pagerank(damping = 0.85, max_iterations = 100L)

# Degree and betweenness centrality
deg <- client$run_degree_centrality(direction = "both")
btw <- client$run_betweenness_centrality()
```

Each returns a named list mapping the node ID (as a character key) to its
score.

### Community Detection and Components

```{r algorithms-community}
# Louvain community detection
louvain <- client$run_louvain()
cat("Communities found:", louvain$num_communities, "\n")

# Connected components (weakly connected by default; strong = TRUE for SCCs)
cc <- client$run_connected_components(strong = FALSE)
cat("Number of components:", cc$count, "\n")
```

## Traversal, Lookups, and Statistics

### Depth-First Search

```{r traversal-dfs}
# Depth-first traversal (a companion to bfs())
visited <- client$dfs(alice, max_depth = 3L)

# ...as of a point in time
visited_past <- client$dfs_at(alice, max_depth = 3L, timestamp = 1672531200000)
```

### Finding and Bulk-Deleting by Label

```{r lookups}
# All node IDs carrying a label
people <- client$find_by_label("Person")

# All edges of a given type, each as {edge_id, source, target}
knows_edges <- client$find_edge_by_type("KNOWS")

# Bulk-delete every node with a label (and its edges); returns the count
removed <- client$delete_by_label("Temporary")
```

### Subgraph Export and Statistics

```{r subgraph-stats}
# Raw subgraph (nodes + edges) around a center node, for visualization
sg <- client$get_subgraph(alice, hops = 2L, max_nodes = 100L)

# Graph-wide statistics
stats <- client$graph_stats()
cat("Nodes:", stats$total_nodes, " Edges:", stats$total_edges, "\n")
```

## Arrow Flight Transport

For large result sets and analytical workloads, the Arrow Flight transport
provides significantly better performance than JSON/TCP. Arrow Flight uses
Apache Arrow's columnar format for zero-copy data exchange between the server
and R.

### When to Use Arrow Flight

- **Large GQL query results** -- Hundreds of thousands of rows or more.
- **Analytical pipelines** -- When results will be processed with columnar
  tools (dplyr, data.table, Arrow compute).
- **Streaming ingestion** -- Processing results in batches to bound memory
  usage.

For small CRUD operations (create a node, get an edge), JSON/TCP is
perfectly adequate and has no extra dependencies.

### ArrowClient (Direct Use)

The `ArrowClient` class communicates directly with the Arrow Flight endpoint
(default port 7689). It requires the `arrow` package.

```{r arrow-client}
# install.packages("arrow")  # if not already installed
library(AstraeaDB)

ac <- ArrowClient$new("grpc://localhost:7689")
ac$connect()

# Execute a GQL query -- returns an Arrow Table
table <- ac$query("MATCH (p:Person) RETURN p.name, p.age")

# Convert to data.frame
df <- as.data.frame(table)

# Or use the convenience method
df <- ac$query_df("MATCH (p:Person) RETURN p.name, p.age")
```

For very large result sets, stream the results in batches to control memory
usage:

```{r arrow-batches}
ac$query_batches(
  "MATCH (n) RETURN n",
  callback = function(batch) {
    cat("Received batch with", nrow(batch), "rows\n")
    # Process each batch incrementally
  }
)

ac$disconnect()
```

You can also use the convenience wrapper:

```{r arrow-convenience}
ac <- astraea_arrow_connect("grpc://localhost:7689")
# ... work ...
ac$disconnect()
```

### UnifiedClient (Automatic Transport Selection)

The `UnifiedClient` is the recommended choice when you want the best of both
worlds. It delegates CRUD operations to the JSON/TCP client and routes GQL
queries through Arrow Flight when the `arrow` package is installed:

```{r unified-client}
uc <- UnifiedClient$new(
  host       = "127.0.0.1",
  port       = 7687L,
  flight_uri = "grpc://localhost:7689"
)
uc$connect()

# Check which transports are active
uc$is_arrow_enabled()
#> [1] TRUE

# CRUD operations go through JSON/TCP
node_id <- uc$create_node(c("Person"), list(name = "Grace", age = 42))

# Queries go through Arrow Flight (or fall back to JSON/TCP)
df <- uc$query_df("MATCH (p:Person) RETURN p.name, p.age")

# All other operations are available as usual
uc$neighbors(node_id, direction = "outgoing")
uc$vector_search(c(0.5, 0.5, 0.5, 0.5), k = 3L)

uc$disconnect()
```

If the `arrow` package is not installed, or if the Arrow Flight connection
fails, the `UnifiedClient` silently falls back to JSON/TCP for everything.
A message is printed when fallback occurs.

## Authentication

AstraeaDB supports token-based authentication with server-side role-based
access control (RBAC). Three roles are available:

| Role     | Permissions                                       |
|:---------|:--------------------------------------------------|
| Reader   | Read-only: get nodes/edges, traversals, queries.  |
| Writer   | Read and write: CRUD, batch, import/export.       |
| Admin    | Full access: all operations plus server management.|

### Using an Auth Token

Pass the `auth_token` parameter when creating any client. The token is
automatically attached to every request:

```{r auth-token}
# AstraeaClient with authentication
client <- AstraeaClient$new(
  host       = "127.0.0.1",
  port       = 7687L,
  auth_token = "my-secret-token"
)
client$connect()

# All operations now carry the token
client$ping()
client$create_node(c("Person"), list(name = "Secured"))
client$disconnect()
```

The convenience function also accepts a token:

```{r auth-connect}
client <- astraea_connect(auth_token = "my-secret-token")
# ... work ...
client$disconnect()
```

The `UnifiedClient` passes the token to the JSON/TCP client:

```{r auth-unified}
uc <- UnifiedClient$new(
  host       = "127.0.0.1",
  port       = 7687L,
  auth_token = "my-secret-token"
)
uc$connect()
# ... work ...
uc$disconnect()
```

### Handling Authentication Errors

If the token is missing, invalid, or the user's role lacks the required
permissions, the server returns an error. Handle it with `tryCatch()`:

```{r auth-error}
tryCatch(
  {
    client <- astraea_connect(auth_token = "wrong-token")
    client$create_node(c("Test"), list(x = 1))
  },
  error = function(e) {
    message("Auth error: ", conditionMessage(e))
  }
)
```

## Putting It All Together

The following example builds a small knowledge graph with embeddings and
temporal edges, then demonstrates vector search, hybrid search, temporal
queries, and GraphRAG in a single workflow:

```{r full-workflow}
library(AstraeaDB)
client <- astraea_connect()
on.exit(client$disconnect(), add = TRUE)

# --- Build the graph ---
ml   <- client$create_node(c("Topic"), list(name = "Machine Learning"),
                            embedding = c(0.1, 0.8, 0.9, 0.7))
nlp  <- client$create_node(c("Topic"), list(name = "NLP"),
                            embedding = c(0.2, 0.9, 0.7, 0.6))
kg   <- client$create_node(c("Topic"), list(name = "Knowledge Graphs"),
                            embedding = c(0.85, 0.2, 0.25, 0.1))
rag  <- client$create_node(c("Topic"), list(name = "RAG"),
                            embedding = c(0.6, 0.7, 0.5, 0.4))

to_ms <- function(d) as.numeric(as.POSIXct(d, tz = "UTC")) * 1000

client$create_edge(ml,  nlp, "RELATED_TO", weight = 0.9,
                   valid_from = to_ms("2018-01-01"))
client$create_edge(nlp, rag, "ENABLES",    weight = 0.8,
                   valid_from = to_ms("2022-01-01"))
client$create_edge(kg,  rag, "ENABLES",    weight = 0.85,
                   valid_from = to_ms("2020-01-01"))
client$create_edge(ml,  kg,  "RELATED_TO", weight = 0.7,
                   valid_from = to_ms("2015-01-01"))

# --- Vector search ---
cat("== Vector Search ==\n")
vs <- client$vector_search(c(0.15, 0.85, 0.8, 0.65), k = 2L)
for (r in vs) {
  cat(sprintf("  Node %d (dist %.4f)\n", r$node_id, r$distance))
}

# --- Hybrid search ---
cat("\n== Hybrid Search ==\n")
hs <- client$hybrid_search(
  anchor = ml, query_vector = c(0.6, 0.7, 0.5, 0.4),
  max_hops = 2L, k = 3L, alpha = 0.5
)
for (r in hs) {
  cat(sprintf("  Node %d\n", r$node_id))
}

# --- Temporal query ---
cat("\n== Temporal Query (2019) ==\n")
nbrs_2019 <- client$neighbors_at(ml, "outgoing", to_ms("2019-06-01"))
for (n in nbrs_2019) {
  node <- client$get_node(n$node_id)
  cat(sprintf("  %s\n", node$properties$name))
}
# Only "NLP" and "Knowledge Graphs" -- RAG edge did not exist in 2019

# --- GraphRAG ---
cat("\n== GraphRAG ==\n")
answer <- client$graph_rag(
  question  = "How are ML and RAG connected?",
  anchor    = ml,
  hops      = 2L,
  max_nodes = 20L,
  format    = "prose"
)
cat(answer$answer, "\n")
```

## Summary

This vignette demonstrated the advanced features of the AstraeaDB R package:

- **Vector search** finds nodes by embedding similarity.
- **Hybrid search** blends graph proximity with vector similarity.
- **Semantic search** ranks neighbors or walks the graph by concept
  similarity.
- **Temporal queries** let you explore the graph as it existed at any point
  in time.
- **GraphRAG** extracts subgraph context for LLM-powered question answering.
- **Graph algorithms** compute PageRank, Louvain communities, connected
  components, and degree/betweenness centrality server-side.
- **Arrow Flight** enables high-performance columnar data transfer for
  analytical workloads.
- **Authentication** secures access with token-based RBAC.

For basic CRUD operations, traversals, GQL queries, and data-frame helpers,
see `vignette("getting-started")`. For an overview of the package and data
model, see `vignette("introduction")`.