Kane Zhu
d86e64eed3
refactor(bedrock): unify inference profile metadata handling and cleanup
...
- Add unified profile_metadata dictionary for both SYSTEM_DEFINED and APPLICATION inference profiles
- Remove unused region prefix functions and defaultdict import
- Add TEMPERATURE_TOPP_CONFLICT_MODELS set for Claude model parameter conflicts
- Improve model ARN parsing and error handling in profile enumeration
- Consolidate profile metadata storage to enable consistent feature detection
2025-10-16 15:24:02 +08:00
Kane Zhu
b4800c54a0
feat: add prompt caching support for Claude and Nova models
...
Add comprehensive prompt caching support with flexible control options:
Features:
- ENV variable control (ENABLE_PROMPT_CACHING, default: false)
- Per-request control via extra_body.prompt_caching
- Pattern-based model detection (Claude, Nova)
- Token limit warnings (Nova 20K limit)
- OpenAI-compatible response format (prompt_tokens_details.cached_tokens)
Supported models:
- Claude 3+ models (anthropic.claude-*)
- Nova models (amazon.nova-*)
- Auto-detection prevents breaking unsupported models
Implementation:
- System prompts caching via extra_body.prompt_caching.system
- Messages caching via extra_body.prompt_caching.messages
- Non-streaming and streaming modes
- Compatible with reasoning, thinking, and tool calls
2025-10-15 11:03:19 +08:00
Mengxin Zhu
371d11d101
chore: cleanup useless files
2025-09-30 16:08:56 +08:00
Mengxin Zhu
e3ee9a707f
docs: update deployment instructions and enhance ECR push script
2025-09-30 16:06:21 +08:00
Mengxin Zhu
76a3614f17
fix: properly handle tool_use messages in conversation
2025-06-30 00:14:26 +08:00
Meng Xin Zhu
9cc3ea8253
chore: publish templates to s3 in release workflow ( #64 )
2024-10-28 17:36:35 +08:00
Mengxin Zhu
46fb759137
chore: use correct Dockerfile for building lambda image
2024-10-09 23:39:37 +08:00
Mengxin Zhu
326e566105
chore: use arm64 architecture image for lambda
2024-10-09 23:15:10 +08:00
Meng Xin Zhu
c1ee1b4244
chore: add automation script to release images ( #58 )
2024-10-09 18:20:14 +08:00
Kane Zhu