> ## Documentation Index
> Fetch the complete documentation index at: https://docs.shaktistudio.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Dynamic Lora Compilation

## Introduction

### What's New?

This feature enables users to dynamically load multiple **LoRAs (Low-Rank Adaptations)** into a single model deployment. With this enhancement, you can tailor your models to diverse tasks and domains without creating multiple separate deployments. By leveraging multiple LoRAs simultaneously, you can optimize model performance, reduce inference time, and streamline your model management workflows.

## Available Flags & Options

| Flag Name           | Type    | Default | Description                                                                                  |
| ------------------- | ------- | ------- | -------------------------------------------------------------------------------------------- |
| `loras`             | list    | `[]`    | List of LoRA configurations to load. See **Example Configuration** for the schema.           |
| `lora_repo`         | dict    | `null`  | Cloud storage path (e.g. S3, GCP) containing multiple LoRAs to load dynamically.             |
| `load_lora_dynamic` | boolean | `false` | Enables dynamic loading of LoRAs into the base model. `false` will merge all provided LoRAs. |

***

## Example Pipeline Configuration

### Using `loras` list:

When specifying LoRAs manually:

```json theme={null}
{
  "type": "llm",
  "loras": [
    {
      "id": "lora_id_0",
      "source": {
        "path": "algoprog/fact-generation-llama-3.1-8b-instruct-lora",
        "type": "hf",
        "secret": { "type": "hf" }
      }
    },
    {
      "id": "lora_id_1",
      "source": {
        "path": "Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16",
        "type": "hf",
        "secret": { "type": "hf" }
      }
    },
    {
      "id": "lora_id_2",
      "source": {
        "path": "raaec/llama3.1-8b-instruct-lora-model",
        "type": "hf",
        "secret": { "type": "hf" }
      }
    }
  ],
  "lora_repo": {
    "type": "",
    "path": "",
    "ownership": "",
    "secret": { "type": "" }
  },
  "quantized_model_path": {
    "type": "",
    "path": "",
    "ownership": "",
    "secret": { "type": "" }
  },
  "load_lora_dynamic": false
}
```

### Using `lora_repo`:

When pulling LoRAs dynamically from a cloud directory:

```json theme={null}
"pipeline_config": {
  "type": "llm",
  "loras": [],
  "lora_repo": {
    "type": "s3",
    "path": "s3://<your-bucket-or-repo>/<your-path>",
    "ownership": "",
    "secret": { "type": "aws" }
  },
  "quantized_model_path": {
    "type": "",
    "path": "",
    "ownership": "",
    "secret": { "type": "" }
  },
  "load_lora_dynamic": false
}
```

***

## Important Notes

✅ If `load_lora_dynamic` is `false` but the `loras` list contains more than one LoRA, then `load_lora_dynamic` will automatically be set to `true`.\
✅ The `id` specified in each LoRA will be the model's name during inferencing.\
✅ When using a `lora_repo`, each subfolder inside the specified path will become a separate model.\
✅ LoRAs will be dynamically merged or loaded at inference time depending on the `load_lora_dynamic` flag.\
✅ To understand how to structure secrets, refer to the [Secret Management](/model-suite/integrations/secrets) documentation. Here are some sample secrets for LoRAs.

<CodeGroup>
  ```json AWS theme={null}
  { 
      "type": "aws",
      "access_key_id": "<access_key>",
      "secret_access_key": "<secret_key>" 
  }
  ```

  ```json Azure theme={null}
  {
      "type": "azure",
      "auth_json": {
          "client_id": "<client_id>",
          "client_secret": "<client_secret>",
          "subscription_id": "<subscription_id>",
          "tenant_id": "<tenant_id>"
      }
  }
  ```

  ```json GCP theme={null}
  {
      "type": "gcp",
      "auth_json": {
          "auth_provider_x509_cert_url": "<auth_provider_x509_cert_url>",
          "auth_uri": "<auth_uri>",
          "client_email": "<client_email>",
          "client_id": "<client_id>",
          "client_x509_cert_url": "<client_x509_cert_url>",
          "private_key": "<private_key>",
          "private_key_id": "<private_key_id>",
          "project_id": "<project_id>",
          "token_uri": "<token_uri>",
          "type": "<account_type>",
          "universe_domain": "<universe_domain>"
      }
  }
  ```
</CodeGroup>

## Recommendations

* **Performance:** Use `load_lora_dynamic = true` if you want the system to load LoRAs on-demand and minimize startup time.
* **Organizational Structure:** When using a `lora_repo`, name the directories intuitively, as those names will serve as model identifiers.
* **Security:** Configure the appropriate `secret` for S3, GCP, HF, or any supported cloud source to ensure proper authentication and authorization.