AVB/bedrock-access-gateway
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Updated docs (EN only)

Browse Source 
Changes:
* Fixed OpenAI env var names (`OPENAI_API_BASE` --> `OPENAI_BASE_URL`)
* Added openai<1.0.0 example with openai.ChatCompletion
* Corrected typos and fixed formatting
This commit is contained in:
Joao Galego
2024-03-27 12:03:02 +00:00
parent a26b8e6833
commit a81dd84cec
1 changed files with 66 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files

139
README.md
View File

@@ -2,23 +2,17 @@
# Bedrock Access Gateway # Bedrock Access Gateway
OpenAI-Compatible RESTful APIs for Amazon Bedrock OpenAI-compatible RESTful APIs for Amazon Bedrock
## Overview ## Overview
Amazon Bedrock offers a wide range of foundation models (such as Claude 3 Sonnet/Haiku, Llama 2, Mistral/Mixtral etc.) Amazon Bedrock offers a wide range of foundation models (such as Claude 3 Sonnet/Haiku, Llama 2, Mistral/Mixtral, etc.) and a broad set of capabilities for you to build generative AI applications. Check the [Amazon Bedrock](https://aws.amazon.com/bedrock) landing page for additional information.
and a broad set of capabilities for you to build generative AI applications.
Check [Amazon Bedrock](https://aws.amazon.com/bedrock) for more details.
Sometimes, you might have applications developed using OpenAI APIs or SDKs, and you want to experiment with Amazon Sometimes, you might have applications developed using OpenAI APIs or SDKs, and you want to experiment with Amazon Bedrock without modifying your codebase. Or you may simply wish to evaluate the capabilities of these foundation models in tools like AutoGen etc. Well, this repository allows you to access Amazon Bedrock models seamlessly through OpenAI APIs and SDKs, enabling you to test these models without code changes.
Bedrock without modifying your codebase. Or you may simply wish to evaluate the capabilities of these foundation models
in tools like AutoGen etc. Well, this repository allows you to access Amazon Bedrock models seamlessly through OpenAI
APIs and SDKs, enabling you to test these models without code changes.
If you find this GitHub repository useful, please consider giving it a free star to show your appreciation and support If you find this GitHub repository useful, please consider giving it a free star ⭐ to show your appreciation and support for the project.
for the project.
Features: **Features:**
- [x] Support streaming response via server-sent events (SSE) - [x] Support streaming response via server-sent events (SSE)
- [x] Support Model APIs - [x] Support Model APIs
@@ -27,24 +21,22 @@ Features:
- [ ] Support Embedding APIs - [ ] Support Embedding APIs
- [ ] Support Image APIs - [ ] Support Image APIs
> NOTE: 1. The legacy [text completion](https://platform.openai.com/docs/api-reference/completions) API is not > **Important notes:** 1/ The legacy [text completion](https://platform.openai.com/docs/api-reference/completions) API is not
> supported, you should move to chat completion API. 2. May support other APIs such as fine-tuning, Assistants API etc. > supported, you should move to chat completion API. 2/ May support other APIs such as fine-tuning, Assistants API, etc. in the future.
> in the future.
Supported Amazon Bedrock models (Model IDs): Supported Amazon Bedrock models (Model IDs):
- anthropic.claude-instant-v1 - `anthropic.claude-instant-v1`
- anthropic.claude-v2:1 - `anthropic.claude-v2:1`
- anthropic.claude-v2 - `anthropic.claude-v2`
- anthropic.claude-3-sonnet-20240229-v1:0 - `anthropic.claude-3-sonnet-20240229-v1:0`
- anthropic.claude-3-haiku-20240307-v1:0 - `anthropic.claude-3-haiku-20240307-v1:0`
- meta.llama2-13b-chat-v1 - `meta.llama2-13b-chat-v1`
- meta.llama2-70b-chat-v1 - `meta.llama2-70b-chat-v1`
- mistral.mistral-7b-instruct-v0:2 - `mistral.mistral-7b-instruct-v0:2`
- mistral.mixtral-8x7b-instruct-v0:1 - `mistral.mixtral-8x7b-instruct-v0:1`
> Note: The default model is set to `anthropic.claude-3-sonnet-20240229-v1:0`. You can change it via Lambda environment > **Note:** The default model is set to `anthropic.claude-3-sonnet-20240229-v1:0` which can be changed via Lambda environment variables (`DEFAULT_MODEL`).
> variables.
## Get Started ## Get Started
@@ -54,30 +46,23 @@ Please make sure you have met below prerequisites:
- Access to Amazon Bedrock foundation models. - Access to Amazon Bedrock foundation models.
If you haven't got model access, please follow > For more information on how to request model access, please refer to the [Amazon Bedrock User Guide](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html) (Set Up > Model access)
the [Set Up](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html) guide
### Architecture ### Architecture
The following diagram illustrates the reference architecture. Note that it also includes a new **VPC** with two public The following diagram illustrates the reference architecture. Note that it also includes a new **VPC** with two public subnets only for the Application Load Balancer (ALB).
subnets only for the Application Load Balancer (ALB).
![Architecture](assets/arch.svg) ![Architecture](assets/arch.svg)
> Note: You can use Lambda Web Adapter + Function URL ( > Note: You can use Lambda Web Adapter + Function URL (see [example](https://github.com/awslabs/aws-lambda-web-adapter/tree/main/examples/fastapi-response-streaming) to replace ALB or AWS Fargate to replace Lambda to get better performance on streaming response.
> See [Example](https://github.com/awslabs/aws-lambda-web-adapter/tree/main/examples/fastapi-response-streaming))
> to replace ALB or AWS Fargate to replace Lambda to get better performance on streaming response.
### Deployment ### Deployment
Please follow below steps to deploy the Bedrock Proxy APIs into your AWS account. Only support regions where Amazon Please follow the steps below to deploy the Bedrock Proxy APIs into your AWS account. Only supports regions where Amazon Bedrock is available (such as `us-west-2`). The deployment will take approximately **3-5 minutes** 🕒.
Bedrock is available (such as us-west-2). The deployment will take approximately 3-5 minutes.
**Step 1: Create you own custom API key (Optional)** **Step 1: Create you own custom API key (Optional)**
> NOTE: This step is to use any string (without spaces) you like to create a custom API Key (credential) that will be > **Note:** This step is to use any string (without spaces) you like to create a custom API Key (credential) that will be used to access the proxy API later. This key does not have to match your actual OpenAI key, and you don't need to have an OpenAI API key. It is recommended that you take this step and ensure that you keep the key safe and private.
> used to access the proxy API later. This key does not have to match your actual OpenAI key, and you don't even need to
> have an OpenAI API key. It is recommended that you take this step and ensure that you keep the key safe and private.
1. Open the AWS Management Console and navigate to the Systems Manager service. 1. Open the AWS Management Console and navigate to the Systems Manager service.
2. In the left-hand navigation pane, click on "Parameter Store". 2. In the left-hand navigation pane, click on "Parameter Store".
@@ -101,32 +86,34 @@ Bedrock is available (such as us-west-2). The deployment will take approximately
3. Click "Next". 3. Click "Next".
4. On the "Specify stack details" page, provide the following information: 4. On the "Specify stack details" page, provide the following information:
- Stack name: Change the stack name if needed. - Stack name: Change the stack name if needed.
- ApiKeyParam (if you set up an API key in Step 1): Enter the parameter name you used for storing the API key ( - ApiKeyParam (if you set up an API key in Step 1): Enter the parameter name you used for storing the API key (e.g., `BedrockProxyAPIKey`). If you did not set up an API key, leave this field blank. Click "Next".
e.g., "BedrockProxyAPIKey"). If you did not set up an API key, leave this field blank.
Click "Next".
5. On the "Configure stack options" page, you can leave the default settings or customize them according to your needs. 5. On the "Configure stack options" page, you can leave the default settings or customize them according to your needs.
6. Click "Next". 6. Click "Next".
7. On the "Review" page, review the details of the stack you're about to create. Check the "I acknowledge that AWS 7. On the "Review" page, review the details of the stack you're about to create. Check the "I acknowledge that AWS CloudFormation might create IAM resources" checkbox at the bottom.
CloudFormation might create IAM resources" checkbox at the bottom.
8. Click "Create stack". 8. Click "Create stack".
That is it! Once deployed, click the CloudFormation stack and go to **Outputs** tab, you can find the API Base URL That is it! 🎉 Once deployed, click the CloudFormation stack and go to **Outputs** tab, you can find the API Base URL from `APIBaseUrl`, the value should look like `http://xxxx.xxx.elb.amazonaws.com/api/v1`.
from `APIBaseUrl`, the value should look like `http://xxxx.xxx.elb.amazonaws.com/api/v1`.
### SDK/API Usage ### SDK/API Usage
All you need is the API Key and the API Base URL. And if you didn't All you need is the API Key and the API Base URL. If you didn't set up your own key, then the default API Key (`bedrock`) will be used.
set up your own key, then the default API Key `bedrock` will be used.
Now, you can try out the proxy APIs. Let's say you want to test Claude 3 Sonnet model, then Now, you can try out the proxy APIs. Let's say you want to test Claude 3 Sonnet model (model ID: `anthropic.claude-3-sonnet-20240229-v1:0`)...
use `anthropic.claude-3-sonnet-20240229-v1:0` as the Model ID.
- **Example API Usage** **Example API Usage**
```bash ```bash
curl https://<API base url>/chat/completions \ export OPENAI_API_KEY=<API key>
export OPENAI_BASE_URL=<API base url>
# For older versions
# https://github.com/openai/openai-python/issues/624
export OPENAI_API_BASE=<API base url>
```
```bash
curl $OPENAI_BASE_URL/chat/completions \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-H "Authorization: Bearer <API Key>" \ -H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{ -d '{
"model": "anthropic.claude-3-sonnet-20240229-v1:0", "model": "anthropic.claude-3-sonnet-20240229-v1:0",
"messages": [ "messages": [
@@ -138,12 +125,7 @@ curl https://<API base url>/chat/completions \
}' }'
``` ```
- **Example SDK Usage** **Example SDK Usage**
```bash
export OPENAI_API_KEY=<API key>
export OPENAI_API_BASE=<API base url>
```
```python ```python
from openai import OpenAI from openai import OpenAI
@@ -157,6 +139,19 @@ completion = client.chat.completions.create(
print(completion.choices[0].message.content) print(completion.choices[0].message.content)
``` ```
For older versions (`openai<1.0.0`), using `openai.ChatCompletion`
```python
import openai
completion = openai.ChatCompletion.create(
model="anthropic.claude-3-sonnet-20240229-v1:0",
messages=[{"role": "user", "content": "Hello!"}],
)
print(completion.choices[0].message.content)
```
## Other Examples ## Other Examples
### AutoGen ### AutoGen
@@ -171,6 +166,8 @@ Make sure you use `ChatOpenAI(...)` instead of `OpenAI(...)`
```python ```python
# pip install langchain-openai # pip install langchain-openai
import os
from langchain.chains import LLMChain from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate from langchain.prompts import PromptTemplate
from langchain_openai import ChatOpenAI from langchain_openai import ChatOpenAI
@@ -178,8 +175,8 @@ from langchain_openai import ChatOpenAI
chat = ChatOpenAI( chat = ChatOpenAI(
model="anthropic.claude-3-sonnet-20240229-v1:0", model="anthropic.claude-3-sonnet-20240229-v1:0",
temperature=0, temperature=0,
openai_api_key="xxxx", openai_api_key=os.environ['OPENAI_API_KEY'],
openai_api_base="http://xxx.elb.amazonaws.com/api/v1", openai_api_base=os.environ['OPENAI_BASE_URL'],
) )
template = """Question: {question} template = """Question: {question}
@@ -207,7 +204,7 @@ Short answer is that API Gateway does not support server-sent events (SSE) for s
### Which regions are supported? ### Which regions are supported?
This solution only supports the regions where Amazon Bedrock is available, so: This solution only supports the regions where Amazon Bedrock is available:
- US East (N. Virginia) - US East (N. Virginia)
- US West (Oregon) - US West (Oregon)
@@ -219,36 +216,33 @@ Note that not all models are available in those regions.
### Can I build and use my own ECR image ### Can I build and use my own ECR image
Yes, you can clone the repo and build the container image by yourself (src/Dockerfile) and then push to your ECR repo. Yes, you can clone the repo and build the container image by yourself (`src/Dockerfile`) and then push to your ECR repo.
Replace the repo url in the CloudFormation template before you deploy. Replace the repo url in the CloudFormation template before you deploy.
### Can I run this locally ### Can I run this locally
Yes, you can run this locally, then the API base url should be like `http://localhost:8000/api/v1` Yes, you can run this locally.
The API base url should look like `http://localhost:8000/api/v1`.
### Any performance sacrifice or latency increase by using the proxy APIs ### Any performance sacrifice or latency increase by using the proxy APIs
Comparing with the AWS SDK call, the referenced architecture will bring additional latency on response, you can try and Comparing with the AWS SDK call, the referenced architecture will bring additional latency on response, you can try and test that own you own.
test that own you own.
Also, you can use Lambda Web Adapter + Function URL ( Also, you can use Lambda Web Adapter + Function URL (see [example](https://github.com/awslabs/aws-lambda-web-adapter/tree/main/examples/fastapi-response-streaming)) to replace ALB or AWS Fargate to replace Lambda to get better performance on streaming response.
See [Example](https://github.com/awslabs/aws-lambda-web-adapter/tree/main/examples/fastapi-response-streaming)) to
replace ALB or AWS Fargate to replace Lambda to get better performance on streaming response.
### Any plan to support SageMaker models? ### Any plan to support SageMaker models?
Currently, there is no plan of supporting SageMaker models. This depends on if there are customer asks. Currently, there is no plan to support SageMaker models. This may change provided there's a demand from customers.
### Any plan to support Bedrock custom models? ### Any plan to support Bedrock custom models?
Fine-tuned models and models with Provisioned Throughput are not supported. You can clone the repo and make the Fine-tuned models and models with Provisioned Throughput are currently not supported. You can clone the repo and make the customization if needed.
customization if needed.
### How to upgrade? ### How to upgrade?
If there is no changes on architecture, you can simply deploy the latest image to your Lambda to use the new If there is no changes on architecture, you can simply deploy the latest image to your Lambda to use the new features (manually) without redeploying the whole CloudFormation stack.
features (manually) without redeploying the whole CloudFormation stack.
## Security ## Security
@@ -257,4 +251,3 @@ See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more inform
## License ## License
This library is licensed under the MIT-0 License. See the LICENSE file. This library is licensed under the MIT-0 License. See the LICENSE file.