Required Infrastructure Configuration
| Parameter | Value | Notes |
|---|---|---|
| Accelerator Type | Model-dependent | Select a type based on model size (see below) ✅ Always choose the (any) variant for better availability (if you have selected the managed infrastructure option as your cluster choice) |
What is Tensor Parallelism (TP)?
Tensor Parallelism allows a model’s computation to be split across multiple GPUs or devices, enabling:- Faster inference for large models
- Support for models too large to fit on a single device
- 1 = No tensor parallelism (single GPU)
- 2, 4, 8, etc. = Enable TP across multiple GPUs
You must have at least as many GPUs as the value set in tensor_parallel_size.
Model Mode Types and Compatibility
Different model versions require specific modes to function correctly. When selecting models from Hugging Face, you’ll typically find two variants:- Base models (e.g., meta-llama/Meta-Llama-3-8B) - designed for completion tasks.
- Instruct models (e.g., meta-llama/Meta-Llama-3-8B-Instruct) - optimized for conversational/chat interactions.
- Set mode to embedding when compiling embedding models
- Set mode to chat when compiling conversational/instruct models
- Set mode to completion when compiling base/completion models
Instance Type Selection Guide
Use the table below to guide instance and TP configuration based on your model size:| Model | Recommended Instance Type | Suggested TP | Notes |
|---|---|---|---|
| Gemma 2B | L40s (any) | 1 | Lightweight model; fits on a single GPU |
| LLaMA 3B | L40s (any) | 1 | Also fits on single GPU with headroom |
| Gemma 7B / LLaMA 8B | H100 (any) | 2–4 | Benefits from multi-GPU setup |
| LLaMA 70B | H100 (any) | 2–4+ | Requires high TP and multi-GPU infrastructure |
Common Issues & Fixes
| Issue | Cause | Fix |
|---|---|---|
| 🚫 Job stuck in queue or not scheduled (machine not available) | Accelerator type too specific or unavailable. | ✅ Use the H100 variant of the accelerator type where available to improve scheduling flexibility. |
| 🚫 Out of memory / crashes | Model too large for a single GPU. | ✅ Increase tensor_parallel_size or switch to a higher-memory accelerator type / increase accelerator count. |
| 🚫 TP value ignored or job fails to start | TP set higher than available GPUs. | ✅ Ensure the total number of GPUs provisioned (accelerator count × GPUs per node) is ≥ the configured tensor_parallel_size. |
| 🐢 Slow inference | Underutilized hardware or no parallelism. | ✅ Tune tensor_parallel_size and, where needed, use multi-GPU configurations to better utilize available accelerators. |
🚫 Unsupported model typeError message: The given model path is invalid | The model which you trying to compile is currently not supported by the Shakti Studio platform. | Please raise a support ticket from your Shakti Studio account at Shakti Studio Support, and we will check the feasibility and add support for the model. |
| Mode Types in Pipeline config Model compilation failing even if the correct model and other parameters are selected. | The selected model mode in the pipeline config may be incorrect. | Pipeline Mode Settings: Configure the mode in your pipeline based on your intended use case: - Set mode to embedding when compiling embedding models. - Set mode to chat when compiling conversational/instruct models. - Set mode to completion when compiling base/completion models. The key is ensuring alignment between your model choice and the corresponding pipeline mode configuration. |
Machine Clean-up failedError Message: Cleanup Failure: Exception occured while cleaning up : Error cleaning up Azure resource group | Please raise a support ticket from your Shakti Studio account at Shakti Studio Support, and we will check the reason for the failure. | Please raise a support ticket from your Shakti Studio account at Shakti Studio Support, and we will investigate and help you resolve the issue. |
FAQs
- Can I edit the model name later? No, renaming a model after it’s been added and compiled is not supported. The model name must be set during the initial setup.
- What options are available for model sources? You can choose HuggingFace, where the base model is downloaded directly, or use AWS S3, GCS, or DockerHub by providing the appropriate path (S3 URL, GCS URL, or DockerHub registry link) along with the required credentials so we can retrieve your custom model.
- Do I need authentication keys for external sources? Yes, for model providers like Shakti Cloud S3, GCS, and DockerHub, you must add your authentication keys on the Secrets page in the Shakti Studio platform and use those credentials during the compilation process. For detailed instructions on how to add and manage secrets, see the Secrets documentation.
- Why am I getting a “The given model path is invalid” error? How do I verify if my model path is valid? While the Shakti Studio platform supports most LLM models, this error can occur if a particular model type isn’t supported yet. If you encounter this issue, please raise a ticket from your Shakti Studio account via Shakti Studio Support, and we’ll work on enabling support for your model.
- How do I link my AWS/GCP/Azure account? You will have to add your cloud account details in the integrations section. Refer to this doc on secrets.
- Can I use multiple cloud accounts? Yes, the Shakti Studio platform supports adding and managing multiple cloud accounts.
- How do I choose the right accelerator for my model?
The appropriate accelerator depends largely on the size of your model. Larger models require higher-spec GPUs (and often higher accelerator counts) for optimization and deployment, while smaller models can run efficiently on mid-range accelerators.
- What happens if I run out of quota for GPUs in my Shakti Studio account?
If your Shakti Studio account lacks sufficient GPU quota, the optimization job may fail to start or could get stuck partway through, leading to a failed optimization process.
- What does
accelerator countmean?
Accelerator count refers to the number of GPU instances allocated for a job. For example, if you select H100 as the accelerator and set the count to 2, two H100 machines will be provisioned. This is especially important when the tensor parallelism (TP) value is greater than 1.
- How do I know which accelerator configuration to select?
In Shakti Studio, you primarily choose the accelerator type (for example, NVIDIA A10G, A100, or H100) and the accelerator count for your job. Larger models and higher tensor parallelism values generally benefit from higher-end GPUs and more accelerators, while smaller models can run efficiently on mid-range GPUs with fewer accelerators. Review your model size and TP settings, then pick an accelerator type and count that provide enough GPU memory and throughput for your workload.
- What is quantization and why should I use it? Quantization is the process of reducing the precision of the numbers used to represent a language model’s parameters (e.g., from 32-bit floating point to 8-bit integers) to make the model smaller and faster, with minimal loss in accuracy, helpful for running large language models (LLMs) efficiently
- Which quantization levels are supported?
We support FP16, FP8, and AWQ quantization.
Note that FP8 is not supported on Ampere architecture GPUs like A100 and A10G, as these devices do not natively support FP8 precision.
- Does quantization affect accuracy?
Yes, quantization can result in a slight reduction in model accuracy.