AVB/bedrock-access-gateway
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update docs

Browse Source
This commit is contained in:
Aiden Dai
2024-12-17 17:38:21 +08:00
parent 51bc727b38
commit 581638b794
4 changed files with 92 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
README.md
View File

@@ -6,13 +6,13 @@ OpenAI-compatible RESTful APIs for Amazon Bedrock
## Breaking Changes
This tool can now automatically detect new models supported in Amazon Bedrock.
This solution can now **automatically detect** new models supported in Amazon Bedrock.
So whenever new models are added to Amazon Bedrock, you can immediately try them without the need to wait for code changes to this repo.
This is to use the `ListFoundationModels` api and the `ListInferenceProfiles` api by Amazon Bedrock, due to this change, additional IAM permissions are required to your Lambda/Fargate role.
If you are facing error: 'Unsupported model xxx, please use models API to get a list of supported models' even the model ID is correct,
please either update your existing stack with the new template in the deployment folder or manually add below permissions to the related Lambda/Fargate role.
please either update your existing stack (**Recommended**) with the new template in the deployment folder or manually add below permissions to the related Lambda/Fargate role.
```json
{
@@ -48,19 +48,7 @@ If you find this GitHub repository useful, please consider giving it a free star
Please check [Usage Guide](./docs/Usage.md) for more details about how to use the new APIs.
> **Note:** The legacy [text completion](https://platform.openai.com/docs/api-reference/completions) API is not supported, you should change to use chat completion API.
Supported Amazon Bedrock models family:
- Anthropic Claude 2 / 3 (Haiku/Sonnet/Opus) / 3.5 Sonnet
- Meta Llama 2 / 3
- Mistral / Mixtral
- Cohere Command R / R+
- Cohere Embedding
You can call the `models` API to get the full list of model IDs supported.
> **Note:** The default model is set to `anthropic.claude-3-sonnet-20240229-v1:0` which can be changed via Lambda environment variables (`DEFAULT_MODEL`).
> **Note:** The default model is set to `anthropic.claude-3-sonnet-20240229-v1:0` which can be changed via Lambda environment variables (`DEFAULT_MODEL`). You can call the [Models API](./docs/Usage.md#models-api) to get the full list of model IDs supported.
## Get Started
@@ -224,20 +212,14 @@ Short answer is that API Gateway does not support server-sent events (SSE) for s
### Which regions are supported?
This solution only supports the regions where Amazon Bedrock is available, as for now, below are the list.
- US East (N. Virginia): us-east-1
- US West (Oregon): us-west-2
- Asia Pacific (Singapore): ap-southeast-1
- Asia Pacific (Sydney): ap-southeast-2
- Asia Pacific (Tokyo): ap-northeast-1
- Europe (Frankfurt): eu-central-1
- Europe (Paris): eu-west-3
Generally speaking, all regions that Amazon Bedrock supports will also be supported, if not, please raise an issue in Github.
Note that not all models are available in those regions.
### Which models are supported?
You can use the [Models API](./docs/Usage.md#models-api) to get/refresh a list of supported models in the current region.
### Can I build and use my own ECR image
Yes, you can clone the repo and build the container image by yourself (`src/Dockerfile`) and then push to your ECR repo. You can use `scripts/push-to-ecr.sh`
@@ -246,7 +228,11 @@ Replace the repo url in the CloudFormation template before you deploy.
### Can I run this locally
Yes, you can run this locally.
Yes, you can run this locally, e.g. run below command under `src` folder:
```bash
uvicorn api.app:app --host 0.0.0.0 --port 8000
```
The API base url should look like `http://localhost:8000/api/v1`.

40
README_CN.md
View File

@@ -6,13 +6,13 @@
## 重大变更
这个方案现在可以自动检测 Amazon Bedrock 中支持的新模型。
这个方案现在可以**自动检测** Amazon Bedrock 中支持的新模型。
因此，当 Amazon Bedrock 添加新模型时，您可以立即尝试使用它们，无需等待此代码库的更新。
这是通过使用Amazon Bedrock 的 `ListFoundationModels API` 和 `ListInferenceProfiles` API 实现的。由于这一变更，您需要为 Lambda/Fargate 角色添加额外的 IAM 权限。
如果您遇到错误："Unsupported model xxx, please use models API to get a list of supported models"（即使Model ID 是正确的），
请使用Deployment 文件夹中的新模板更新您现有的堆栈，或手动为相关的 Lambda/Fargate 角色添加以下权限。
请使用Deployment 文件夹中的新模板更新您现有的堆栈(**推荐**），或手动为相关的 Lambda/Fargate 角色添加以下权限。
```json
{
@@ -50,19 +50,7 @@ OpenAI 的 API 或 SDK 无缝集成并试用 Amazon Bedrock 的模型,而无需
请查看[使用指南](./docs/Usage_CN.md)以获取有关如何使用新API的更多详细信息。
> 注意： 不支持旧的 [text completion](https://platform.openai.com/docs/api-reference/completions) API，请更改为使用Chat Completion API。
支持的Amazon Bedrock模型家族：
- Anthropic Claude 2 / 3 (Haiku/Sonnet/Opus) / 3.5 Sonnet
- Meta Llama 2 / 3
- Mistral / Mixtral
- Cohere Command R / R+
- Cohere Embedding
你可以先调用`models` API 获取支持的详细 model ID 列表。
> 注意: 默认模型为 `anthropic.claude-3-sonnet-20240229-v1:0`， 可以通过更改Lambda环境变量进行更改。
> 注意: 默认模型为 `anthropic.claude-3-sonnet-20240229-v1:0`， 可以通过更改Lambda环境变量进行更改。你可以先调用 [Models API](./docs/Usage.md#models-api) 查看支持的详细 model ID 列表。
## 使用指南
@@ -223,20 +211,14 @@ print(response)
### 支持哪些区域?
只支持Amazon Bedrock可用的区域, 截至当前，包括以下区域:
- 美国东部(弗吉尼亚北部)：us-east-1
- 美国西部(俄勒冈州)：us-west-2
- 亚太地区(新加坡)：ap-southeast-1
- 亚太地区(悉尼)：ap-southeast-2
- 亚太地区(东京)：ap-northeast-1
- 欧洲(法兰克福)：eu-central-1
- 欧洲(巴黎)：eu-west-3
通常来说，所有Amazon Bedrock支持的区域都支持，如果不支持，请提个Github Issue。
注意，并非所有模型都在上面区可用。
### 支持哪些模型?
你可以通过[Model API](./docs/Usage_CN.md#models-api) 获取（或更新）当前区支持的模型列表。
### 我可以构建并使用自己的ECR镜像吗?
是的,你可以克隆repo并自行构建容器镜像(src/Dockerfile),然后推送到你自己的ECR仓库。 脚本可以参考`scripts/push-to-ecr.sh`。
@@ -245,7 +227,13 @@ print(response)
### 我可以在本地运行吗?
是的,你可以在本地运行,那么API Base URL应该类似于`http://localhost:8000/api/v1`
是的,你可以在本地运行, 例如在`src` 文件夹下运行：
```bash
uvicorn api.app:app --host 0.0.0.0 --port 8000
```
那么API Base URL应该类似于`http://localhost:8000/api/v1`
### 使用代理API会有任何性能牺牲或延迟增加吗?

34
docs/Usage.md
View File

@@ -9,6 +9,40 @@ export OPENAI_API_KEY=<API key>
export OPENAI_BASE_URL=<API base url>
```
## Models API
You can use this API to get a list of supported model IDs.
Also, you can use this API to refresh the model list if new models are added to Amazon Bedrock.
**Example Request**
```bash
curl -s $OPENAI_BASE_URL/models -H "Authorization: Bearer $OPENAI_API_KEY" | jq .data
```
**Example Response**
```bash
[
...
{
"id": "anthropic.claude-3-5-sonnet-20240620-v1:0",
"created": 1734416893,
"object": "model",
"owned_by": "bedrock"
},
{
"id": "us.anthropic.claude-3-5-sonnet-20240620-v1:0",
"created": 1734416893,
"object": "model",
"owned_by": "bedrock"
},
...
]
```
## Embedding API
**Important Notice**: Please carefully review the following points before using this proxy API for embedding.

32
docs/Usage_CN.md
View File

@@ -9,6 +9,38 @@ export OPENAI_API_KEY=<API key>
export OPENAI_BASE_URL=<API base url>
```
## Models API
你可以通过这个API 获取支持的models 列表。 另外，如果Amazon Bedrock有新模型加入后，你也可以用它来更新刷新模型列表。
**Request 示例**
```bash
curl -s $OPENAI_BASE_URL/models -H "Authorization: Bearer $OPENAI_API_KEY" | jq .data
```
**Response 示例**
```bash
[
...
{
"id": "anthropic.claude-3-5-sonnet-20240620-v1:0",
"created": 1734416893,
"object": "model",
"owned_by": "bedrock"
},
{
"id": "us.anthropic.claude-3-5-sonnet-20240620-v1:0",
"created": 1734416893,
"object": "model",
"owned_by": "bedrock"
},
...
]
```
## Embedding API
**重要**: 在使用此代理 API 之前,请仔细阅读以下几点: